Bringing a Modern Javascript Build Pipeline to LinkedIn

(Photo by surfglassy, shared via Creative Commons license)

Co-author: Adam Miller

At LinkedIn, our system for managing shared static content in apps was becoming unscalable as we turned the corner from 2014 to 2015. Up until then, our shared Web front end assets consumed across LinkedIn products had been built and deployed as a single monolithic package. It had been under stress since we moved to an object-oriented approach for writing shared CSS and Javascript. However, we really began to feel the limits of this method when we started writing Javascript Single Page Apps, which introduced a need for more granular dependency management. These are some scenarios that showed we were near the end of the lifespan for the monolithic approach:

  1. A single bad commit could invalidate the entire monolith, blocking multiple front end services that were bound to the bad version.
  2. There was no useful graph of Javascript dependencies to guide us when making design or coding decisions.
  3. Lack of granular dependency management resulted in long page loads due to unnecessarily large JavaScript packages.
  4. Engineers created ugly hacks to try and retrofit this legacy model with new, industry-standard practices like ES6 transpiling, module bundling and new SCSS features.
  5. Even minor changes, like a single pixel, required lengthy rebuilds and releases of the monolith and its dependents, slowing developer productivity and discouraging engineers from contributing and consuming shared static code.

Migrating LinkedIn away from this monolithic stack of static code has been a herculean, cross-team effort. I am excited to describe some of the changes we’ve made to improve the front end dependency ecosystem here at LinkedIn.

 

Broccoli and NPM

We wanted to upgrade our capabilities for not only consuming static content, but also open up our pipeline to the many new and powerful static content build tools that are now standard throughout the open source web development communities. Additionally, we wanted to integrate the new pipeline with our existing Gradle infrastructure. Enter Node Package Manager (NPM) and Broccoli.

NPM is the de facto standard for JavaScript package management. It gives developers amazing granularity and control over their versioned packages. In addition, the vast majority of new web developer tooling is written for NPM and its associated tools. Adopting NPM as our package manager for static content and build tools allows us to  tap into the work and talent of thousands of open source developers.

There are many JavaScript build tools to choose from, but the recent newcomer, Broccoli, suited our needs the best. Unlike other task-centric build systems like Grunt or Gulp, Broccoli consists exclusively of JavaScript files whose main export is a transformed file tree – a “Brocfile.” Any other Brocfile may then import, operate on, and re-export this file tree. This allows us to easily execute pre and post steps on user-provided Brocfiles, abstracting away any LinkedIn-specific magic that needs to happen. Additionally, Broccoli allows us to deliver through NPM and automatically re-compile SCSS code via Eyeglass, transpile our application code from ES6 to ES5 using Babel, bundle our JavaScripts using tools like Browserify and Webpack, run jsHint, better automate our front end testing, and much more.

By adopting these open source and industry standard tools into the LinkedIn ecosystem, our front end developers are given intimate control over the lifecycle of their application code. However, as anyone who works with systems at scale will tell youthere will be friction when trying to integrate with and move any large system, regardless of how willing the users may be.

Integration

There is never a one-size-fits-all solution for major upgrades, and we have taken that to heart while integrating NPM and Broccoli into our developer ecosystem. As such, we provide two upgrade paths to our developers.

Upgrade Path One: Embracing the Ember Ecosystem

The first requires switching to the entire Ember ecosystem, which is the single page Web app framework we decided to standardize on a year ago. There are two variants of the Ember ecosystem we needed to supportEmber with Play as the mid-tier (internally known as Pemberly), and Ember with Python+Flask as the mid-tier (internally known as Flyer). For both of these ecosystems, we use Ember-CLI (which is powered by NPM and Broccoli under the hood) to build the projects, with a little LinkedIn “special sauce” to make the output work with our i18n and CDN infrastructures. The entire output directory is then hosted on our existing app servers. With Ember, we gain the support of a very active open source community. Its opinionated style of coding allows for better interoperability between teams as engineers change roles. Ember also lets engineers focus on writing great application code, and leaves the time-consuming infrastructure work to the open source community at large. In fact, some of our LinkedIn engineers have been busy contributing to the overall ecosystem: Chad Hietala, Nathan Hammond, Chris Eppstein, Kris Selden, Eugene O’Neill, and Asa Kusuma.

This approach works wonderfully, as long as the team adopting the Ember ecosystem has slated a time-consuming rebuild into their product timelines. Unfortunately, this is rarely possible in the short term as it is hard to justify a re-write that does not coincide with a significant update of the user experience. In order for the 100+ existing Play+Dust stacks here at LinkedIn to benefit without making the leap to Ember, and to garner quick, widespread adoption across the company, we needed a second-tier option that could deliver the new features of an NPM static asset pipeline without the high adoption costs.

Upgrade Path Two: The Play-NPM-Pipeline

We call this tier-2 option the Play-NPM-Pipeline. It is a method for existing front ends to easily shim a modern NPM workflow into an existing Play stack. Play-NPM-Pipeline is just another node module and delivers three different tools:

  1. When installed globally it delivers a command line tool that can upgrade any LinkedIn Play server to use NPM and the updated tooling.
  2. When installed locally in a project it provides commands for “npm start,” “npm run build,” and “npm run clean” to help run the new Node builds.
  3. When required as a Node package it makes a number of tools available to help devs customize their Broccoli build script.

Let’s dive in and see what these three components deliver to devs.

1) The Play-NPM-Pipeline Upgrade

Using the command line upgrade script, any Play stack at LinkedIn can be converted to use NPM and consume node modules. Three major changes happen during an upgrade:

A.    A package.json and Brocfile.js is added to the root of the project, with happy defaults set for the Play-NPM-Pipeline infrastructure.

B.    We maintain backward compatibility with the old directory structure while creating a new structure for Javascript builds.

C.    A few new Gradle tasks appear in the project. These are called by Play at certain times in a Play app’s lifecycle (run, build, clean, etc.) and in turn call out to the respective NPM commands.

2) The Play-NPM-Pipeline Runner

Because we now have hooks into the major lifecycle events of a Play app, Play-NPM-pipeline can insert JavaScript build tasks anywhere they need to go. The new package.json in an upgraded project comes pre-installed with a dependency on the play-npm-pipeline package, and the default “npm start,” “npm build,” and “npm clean” commands are set to the corresponding scripts delivered by it.

During a build, we run the project’s new Brocfile.js on the contents of the /assets folder and place the build output in /public. From there, Play’s file watchers take over. Because the output of the Broccoli pipeline is just properly formatted front end code, Play can serve up the files exactly as it had before.

The Broccoli build step is run only once during a Play build, so if a developer is starting up the project to make back end changes they don’t need to know anything about the new pipeline before getting to workeverything functions as it had before. If a developer wants to make front end changes, they open a new terminal window and run an “npm start” to kick off the Broccoli file watcher. As they modify files, the pipeline will re-run automatically, handing off the re-compiled files to Play for it to serve to the browser. Running the Broccoli watcher in a separate terminal has the added benefit of keeping front end build console output separate from the Play server’s console output.

3) The Play-NPM-Pipeline Build

Everything that happens during a front end build lives in the project’s new Brocfile.js. Broccoli allows projects to customize their build to suit their needs by importing any Broccoli plugin and applying it to their project. Play-NPM-Pipeline makes this job even easier by providing special tools that work out of the box with LinkedIn projects. If a dev uses “require(play-npm-pipeline);” in their Brocfile, they are delivered a Broccoli tree of their “/assets” directory, a SASS compiler, module bundler, ES6 transpiler, and more.

The out-of-the-box Brocfile does absolutely nothing. That’s right, nothing. Line one requires Play-NPM-Pipeline and line two exports “/assets” as is. The intent is to allow projects to upgrade and then add extra features as needed. This minimizes the barriers for entry to the new system and ensures we can easily keep the org moving forward together as a whole.

The modern tooling available in this file is what makes the Play-NPM-Pipeline so powerful. By bundling with WebPack, projects can deliver and consume shared JavaScript code through NPM. Eyeglass, a next generation SCSS compiler, allows the company to deliver shared styling libraries, icons, fonts and images via an NPM module, and we’re working to upgrade our template compiler, Dust, to deliver templates through NPM as well. By making these tools available we can move away from our monolithic static content package quickly and in a way that offers agility for adopting teams.

The Future

This is just the first step toward many more improvements in how we do front end shared code at LinkedIn. I’d like to thank Adam Miller, Marius Seritan, Eugene O’Neill and Chris Eppstein, who drove this solution to completion, as well as the teams who’ve been helping us drive adoption across the company.