January 20, 2016
- A single bad commit could invalidate the entire monolith, blocking multiple front end services that were bound to the bad version.
- Engineers created ugly hacks to try and retrofit this legacy model with new, industry-standard practices like ES6 transpiling, module bundling and new SCSS features.
- Even minor changes, like a single pixel, required lengthy rebuilds and releases of the monolith and its dependents, slowing developer productivity and discouraging engineers from contributing and consuming shared static code.
Migrating LinkedIn away from this monolithic stack of static code has been a herculean, cross-team effort. I am excited to describe some of the changes we’ve made to improve the front end dependency ecosystem here at LinkedIn.
We wanted to upgrade our capabilities for not only consuming static content, but also open up our pipeline to the many new and powerful static content build tools that are now standard throughout the open source web development communities. Additionally, we wanted to integrate the new pipeline with our existing Gradle infrastructure. Enter Node Package Manager (NPM) and Broccoli.
By adopting these open source and industry standard tools into the LinkedIn ecosystem, our front end developers are given intimate control over the lifecycle of their application code. However, as anyone who works with systems at scale will tell you—there will be friction when trying to integrate with and move any large system, regardless of how willing the users may be.
There is never a one-size-fits-all solution for major upgrades, and we have taken that to heart while integrating NPM and Broccoli into our developer ecosystem. As such, we provide two upgrade paths to our developers.
Upgrade Path One: Embracing the Ember Ecosystem
The first requires switching to the entire Ember ecosystem, which is the single page Web app framework we decided to standardize on a year ago. There are two variants of the Ember ecosystem we needed to support—Ember with Play as the mid-tier (internally known as Pemberly), and Ember with Python+Flask as the mid-tier (internally known as Flyer). For both of these ecosystems, we use Ember-CLI (which is powered by NPM and Broccoli under the hood) to build the projects, with a little LinkedIn “special sauce” to make the output work with our i18n and CDN infrastructures. The entire output directory is then hosted on our existing app servers. With Ember, we gain the support of a very active open source community. Its opinionated style of coding allows for better interoperability between teams as engineers change roles. Ember also lets engineers focus on writing great application code, and leaves the time-consuming infrastructure work to the open source community at large. In fact, some of our LinkedIn engineers have been busy contributing to the overall ecosystem: Chad Hietala, Nathan Hammond, Chris Eppstein, Kris Selden, Eugene O’Neill, and Asa Kusuma.
This approach works wonderfully, as long as the team adopting the Ember ecosystem has slated a time-consuming rebuild into their product timelines. Unfortunately, this is rarely possible in the short term as it is hard to justify a re-write that does not coincide with a significant update of the user experience. In order for the 100+ existing Play+Dust stacks here at LinkedIn to benefit without making the leap to Ember, and to garner quick, widespread adoption across the company, we needed a second-tier option that could deliver the new features of an NPM static asset pipeline without the high adoption costs.
Upgrade Path Two: The Play-NPM-Pipeline
We call this tier-2 option the Play-NPM-Pipeline. It is a method for existing front ends to easily shim a modern NPM workflow into an existing Play stack. Play-NPM-Pipeline is just another node module and delivers three different tools:
- When installed globally it delivers a command line tool that can upgrade any LinkedIn Play server to use NPM and the updated tooling.
- When installed locally in a project it provides commands for “npm start,” “npm run build,” and “npm run clean” to help run the new Node builds.
- When required as a Node package it makes a number of tools available to help devs customize their Broccoli build script.
Let’s dive in and see what these three components deliver to devs.
1) The Play-NPM-Pipeline Upgrade
Using the command line upgrade script, any Play stack at LinkedIn can be converted to use NPM and consume node modules. Three major changes happen during an upgrade:
A. A package.json and Brocfile.js is added to the root of the project, with happy defaults set for the Play-NPM-Pipeline infrastructure.
C. A few new Gradle tasks appear in the project. These are called by Play at certain times in a Play app’s lifecycle (run, build, clean, etc.) and in turn call out to the respective NPM commands.
2) The Play-NPM-Pipeline Runner
During a build, we run the project’s new Brocfile.js on the contents of the /assets folder and place the build output in /public. From there, Play’s file watchers take over. Because the output of the Broccoli pipeline is just properly formatted front end code, Play can serve up the files exactly as it had before.
The Broccoli build step is run only once during a Play build, so if a developer is starting up the project to make back end changes they don’t need to know anything about the new pipeline before getting to work—everything functions as it had before. If a developer wants to make front end changes, they open a new terminal window and run an “npm start” to kick off the Broccoli file watcher. As they modify files, the pipeline will re-run automatically, handing off the re-compiled files to Play for it to serve to the browser. Running the Broccoli watcher in a separate terminal has the added benefit of keeping front end build console output separate from the Play server’s console output.
3) The Play-NPM-Pipeline Build
Everything that happens during a front end build lives in the project’s new Brocfile.js. Broccoli allows projects to customize their build to suit their needs by importing any Broccoli plugin and applying it to their project. Play-NPM-Pipeline makes this job even easier by providing special tools that work out of the box with LinkedIn projects. If a dev uses “require(play-npm-pipeline);” in their Brocfile, they are delivered a Broccoli tree of their “/assets” directory, a SASS compiler, module bundler, ES6 transpiler, and more.
The out-of-the-box Brocfile does absolutely nothing. That’s right, nothing. Line one requires Play-NPM-Pipeline and line two exports “/assets” as is. The intent is to allow projects to upgrade and then add extra features as needed. This minimizes the barriers for entry to the new system and ensures we can easily keep the org moving forward together as a whole.
This is just the first step toward many more improvements in how we do front end shared code at LinkedIn. I’d like to thank Adam Miller, Marius Seritan, Eugene O’Neill and Chris Eppstein, who drove this solution to completion, as well as the teams who’ve been helping us drive adoption across the company.