Developing Play Applications using Gradle

At LinkedIn, we are always looking for the best software development frameworks and tools to build great products. During our 11-year history we have adopted many web frameworks for UI development - among them Grails, Frontier (Linkedin's internal web framework), and most recently, Play. We love Play and are excited to increase our adoption of it across the company. As we evolve and scale our integration with Play, we are now evaluating which build system to use going forward.

This article introduces Gradle's plan of Play support, the current status and our preliminary evaluations of this incubating feature. We would like to involve the Play community to provide more feedbacks to this ongoing work by Gradleware. Being able to build/develop Play applications on both SBT and Gradle shall give Play projects more freedom and versatility.

Background: Code Organization

At LinkedIn we organize repositories around public APIs and services. We call each such repository a Multiproduct. Each Multiproduct is represented by a unique Ivy organization, which defaults to com.linkedin.<product name>. This code organization is apparent in LinkedIn projects that have recently been open sourced, for example databus.

A consequence of organizing code in Multiproducts is that they are all multi-project builds. Each Multiproduct has 10s or 100s of projects, that each produces one or more artifacts.

At LinkedIn, our Play applications have 500+ jars on its classpath, which means a fully resolved dependency graph of 500+ nodes. While resolving this dependency graph we rely heavily on conflict resolution rules. Instead of resolving versions at an artifact (i.e. JAR) level, we resolve them at the Multiproduct (or Ivy organization) level. This ensures we end up with consistent versions of all artifacts from the same Multiproduct.

For example, for Multiproduct util, we have 3 different modules: com.linkedin.util:foo, com.linkedin.util:bar, com.linkedin.util:baz. Each of the modules may have one or more multiple artifacts. Normal conflict resolution ensures only one version of all modules would be used. But the versions of different modules (in this case, "com.linkedin.tuil:foo", "com.linkedin.util:bar", and "com.linkedin.util, baz") may still be different. Our custom dependency resolution will make sure all artifacts from the same Multiproduct are of the same version.

Scalability challenges with SBT

The default build system for Play is SBT. Play and SBT are technically de-coupled, but in the minds of many are inseparable. As we started building Play applications at LinkedIn, we quickly ran into scalability challenges with SBT and the underlying Apache Ivy library. Dependency resolution for our applications can take 5-10 minutes, and in one of our infrastructure projects it takes 30+ minutes.

For the past year, LinkedIn has partnered with Typesafe to invest in improving SBT scalability and we continue to do so. However, at the same time we feel that it is worthwhile to explore other build systems for Play.

Integrating Play and Gradle

LinkedIn has been using Gradle for the past 3.5 years. We use Gradle in everything from small projects to a single multi-project build with nearly 4,000 projects. There are many exciting features coming in 2015 for Gradle.

Gradle Roadmap

In addition to the above, we believe that an integration between Play and Gradle will add significant value to both technologies. Together with Gradleware we are exploring how to make this a reality. In the first half of 2015 we are going to focus on implementing and testing this integration, which will be spread out over 3 milestones.

Milestone 1: Using Gradle to build a Play application

Using a Gradle “play” plug-in, a developer can build a fully featured Play application in this milestone.

Features include:

• Compiling Java and Scala sources
• Compiling Routes
• Compiling Templates
• Compiling Assets
  • CoffeeScript, LESS/CSS, JavaScript

Using this build pipeline, the Play application can be run with gradle run or gradle start, or a distribution created with gradle stage or gradle dist.

Having this functionality will allow us to thoroughly test Play build performance and scalability when Gradle is used as the build system.

Note that at this milestone continuous mode and hot reload won’t be functional.

Milestone 2: Continuous Mode and Hot-Reload

Being able to hot-reload during development is one of our most beloved features of Play.

Automating the manual process of interacting with the build system is a great idea in general, and a story that we are very excited about adding to Gradle. Milestone 2 will introduce a “watcher” mode feature to Gradle, which is equivalent to the continuous mode in SBT: gradle --watch run

Play also has the ability to trigger a rebuild when the browser is refreshed. This is done through a BuildLink interface in Play, which will be supported by Gradle in this milestone.

Milestone 3: Making it real

If you’re familiar with Play you may have noticed a big missing feature: the Scala interactive REPL! Don’t worry…it’s part of Milestone 3, which would allow you to interact with the classes in the interactive console.

And we can’t forget documentation, integration with Specs2, and Scala code quality plug-ins. To tie it all together you will be able to get a bootstrapped Play application with the appropriate Gradle infrastructure, just like you do with activator.

Interested in learning more? You can see details about the technical design at: https://github.com/gradle/gradle/blob/master/design-docs/play-support.md.

Getting a taste of Milestone 1

Milestone 1 is now feature complete, and you can download the latest Gradle nightly build to get a taste of it. The complete distribution contains the binaries, sources, and sample projects. Unzip it somewhere convenient, and put its “bin” directory on your path.

Check out the Play samples in “samples/play” directory.

Basic

The basic sample build.gradle file demonstrates the Play plugin declaration as well as the required repository configurations. The dependencies block shows how to include a 3rd party dependency. The 'play' configuration provides compile/runtime dependencies, and the 'playTest' configuration provides dependencies that are only used in tests.

Advanced

The advanced sample build.gradle file shows how to configure the target Play/Scala version.

Multiproject:

This is a standard Gradle multiproject build. The root project contains the main play application, with 3 submodules in the 'modules' directory. The 'admin' and 'user' submodules are Play applications in themselves whereas the 'util' submodule produces a Java library. The settings.gradle file wires in the 3 subprojects and the root build.gradle file declares project dependencies on each of the 3 submodules.

You can try the following tasks in the sample projects:
- gradle :assemble : builds the application jar and the application assets jar
- gradle :runPlayBinary : builds and runs the application in dev mode
- gradle :check : executes the application tests
- gradle :stage : build the distribution image, with startup scripts

Early Build Performance Evaluation

We compared the clean-build time and incremental build time of building Play projects with both Gradle and Activator (sbt).

Projects Used

• Basic - a simple single project Play application, based on what is generated by activator when choosing the play-scala template.
• Multiproject - a multiproject build based on the gradle’s multiproject Play sample: 3 Scala Play applications as well as a Java util subproject.

Methodology Used

• All dependency libraries were already downloaded to local gradle and ivy cache.
• Testing task “dist”, which would produce the final distribution tarball. This is a test of the entire build and release cycle as this task encompasses dependency resolution, compilation and packaging.
• Gradle was run with parallel, configure-on-demand and daemon enabled whereas vanilla sbt was used.
• Clean-build: showcases the entire build process from a clean slate.
• No change: rerun “dist” without any change. This is to test how quickly the build system configures and detects that everything is up-to-date.
• Incremental: one line change was given to either a Java or Scala subproject that the root Play application depends on to emulate a typical development cycle.

Results (average across 3 runs, in seconds)

Basic       Gradle             Activator (sbt)      

Clean build	18.5	22.2
No change	1.5	9.8

Multiproject       Gradle             Activator (sbt)      

Clean build	24.5	23.7
No change	2.1	11.2

Incremental       Gradle             Activator (sbt)      

Basic (Scala change)	11.5	16
Multiproject (Scala change)	12	19.5
Multiproject (Java change)	6.7	19

From above, we can see build time of Gradle won all categories except one, and its results with no or small changes are especially good, which is exactly what we were expecting.

Preliminary Dependency Resolution Performance Evaluation

We are interested in the dependency resolution time in particular. So we wanted to compare only how long it may take to resolve all the dependent libraries. For activator/sbt it is easy because they delegated the dependency resolution to ivy via the “update” task. However, Gradle doesn’t have a task that would do exactly the same. Luckily Gradle’s built-in profiling feature would show us the dependency resolution time. Also for a more fair comparison, we also measured the entire Gradle execution time.

Project Used

A small Linkedin Play application which consists of 4 subprojects: 2 Java subprojects and two Play subprojects. Please note, unlike the previous test, building this Linkedin Play project requires many Linkedin internal plugins for both Gradle and sbt. For instance, as mentioned above, we leverage our own dependency resolution rules to do a) Organization/group level dependency resolution and b) Dependency Validation: checking for deprecated dependencies. They shall incur extra effort during dependency resolution, but that happened on both Gradle and sbt.

Methodology Used

• All dependency libraries were already downloaded to local gradle and ivy cache.
• For activator/sbt, we measured the time of “update” task that updates all four subprojects. We collected results both with and without sbt console loading.
• For Gradle, we measure the time of “dependencies” task for all four subprojects with built-in profiler enabled.

Results (average across 3 runs, in seconds)

   Average Time   

“update” task only (in sbt console)	31.3
Play update overall (from CLI)	38.7
Reported dependency resolution time	4.75
Gradle dependencies tasks overall	19.55

Notes:

1. Gradle “dependencies” tasks did more than dependency resolution, it actually printed out the dependency tree of all four subprojects.
2. Gradle’s dependency resolution time was self-reported via its built-in profiler.

From the results, we can easily tell Gradle’s native dependency resolution engine is 10x faster than the ivy resolution engine used by activator/sbt. And that is in line with our experience of building other Java projects on Gradle.

We are still working on making Gradle work for our big activator/sbt projects whose dependency resolution would take 5~10 minutes. We are expecting even better results there!

We’d love to hear from you

Contributing to open source technologies such as Play and Gradle are core to LinkedIn’s Engineering mission. If you are using Play or Gradle, we’d love to hear what you think about a potential integration. And if you’re interested in helping us make these and other tools and frameworks awesome for everyone, drop me an inMail.

Acknowledgments

I would like to thank Jens Pillgram-Larsen, Dustin Kwong, Deep Majumder, Bryan Barkley, and Ruobing Li for their contributions to this post.