Great Tools for Engineers: Refactoring Across Multiple Code Bases with Gradle and IntelliJ IDEA

January 20, 2017

  • Multi-repo development

LinkedIn engineers require tooling that scales really well, and we never stop improving it. Even at a smaller scale, providing great tools for engineers is key to winning business and retaining top talent. This post is about working with code that lives in many separate code repositories, while still being productive and efficient in the process!

Repository and SCM agnostic development at LinkedIn

At LinkedIn, we want to create a development environment where the underlying SCM (source control management) system becomes an implementation detail. We built an abstraction layer on top of the source code repositories called “multiproduct.” Think of a multiproduct layer as a code base that is a unit of building, testing, and releasing some software, such as a web app, a service, or a set of software libraries. Multiproducts at LinkedIn can be checked out, and a new feature can be implemented and submitted for review, without interacting directly with the SCM system. A software project, wrapped with multiproduct abstraction, can have the source code stashed in multiple separate code repositories. At the moment, this is a standard scenario for certain kinds of projects at LinkedIn, like services that keep configs in a separate repository or open source wrapper projects that keep public code in GitHub. Multiproduct abstraction does not completely override interactions with SCM system. On a daily basis, engineers rebase their branches, work with revision history, and interact with SCM system in other ways. More about the multiproduct concept will be covered in future articles. Jens Pillgram-Larsen’s post “Find the Seams” is a great introduction.

LinkedIn's multiproduct abstraction presents a boundary between code bases. The challenge emerges when a code change needs to span this boundary. For example, when a common library developed in a separate multiproduct changes its API, clients that use it (have a binary dependency on the library) need to be updated. So, there must be a change in the library code (multiproduct A) and in the client code (multiproduct B). This generates more overhead: one change in the library requires the new version to be published, and then another change is needed in the client code so that it uses the new version of the library. The result is more independent changes, manual ordering work, more CI builds, and binary version management.

There are, however, benefits of the boundary between multiproducts. Working with software components integrated at a binary level via dependency management enables teams to move faster by making it possible to ship ambitious, incompatible changes without the need to update all consumers at once. In multiproduct, thoughtful design of the public API is a necessary practice, and this positively affects the overall architecture. Each team needs to continually care about dependency management to avoid having a complicated dependency graph, unnecessary dependencies, and dependency cycles. It is more work at first, but it pushes the teams harder to keep the architecture clean and healthy.

  • LinkedIn Engineers at a bootcamp

LinkedIn engineers collaborating on a project at one of our regular developer bootcamps

Open source partnerships for a better build experience

At LinkedIn, we want it all: a clean architecture of software components integrated at the binary level and the convenience of cross-multiproduct refactoring. In order to deliver this capability, we sponsored a new feature in the Gradle build system called Composite Builds. In addition, the LinkedIn Development Tools team updated our custom IntelliJ IDEA plugin to take advantage of Gradle Composite Builds and deliver a great developer experience.

Gradle is a robust and extensible open source build platform. At LinkedIn, Gradle serves as the core of the build automation framework for all software stacks we support: JVM, Android, iOS, C++, JavaScript/Web, Python, and more. We recently open sourced support for Python with Gradle—try it out and enjoy Gradle’s amazing dependency management in the Python world! Driving automation of major technology stacks at LinkedIn using the same build tool gives engineers a consistent developer experience. It also reduces the maintenance cost of the build infrastructure by reusing our custom LinkedIn Gradle plugins.

At LinkedIn, we leverage our open source partnership with Gradle Inc. to sponsor new features in Gradle. The partnership gives LinkedIn an opportunity to further enhance our internal build automation experience while adding value to the entire Gradle community. The new features become part of the standard Gradle distribution, free and open sourced for the community to use.

The Gradle Composite Builds feature “connects” separate Gradle builds into one single, cohesive build. This means that we can treat separate multiproducts  as a single multiproduct, built with a single Gradle invocation. A critical feature of Composite Builds is the ability to integrate with IntelliJ IDEA—a very popular IDE. We can import separate multiproducts into a single IntelliJ IDEA window. This allows cross-repository IDE experience: very convenient code navigation and cross-multiproduct refactoring.

To maximize the developer experience at LinkedIn, we have added support for cross-multiproduct development into the LinkedIn IntelliJ IDEA plugin. An IDE is the place where engineers spend most of their time, so adding features into the IDE has huge opportunities to boost engineering productivity at a large organization. We recognize this at LinkedIn, so custom IDE integration via plugins is a key component of our engineering tooling roadmap. If you want to see a demo of how cross-multiproduct refactoring is implemented at LinkedIn, check out the first seven minutes of this video from the Gradle meetup LinkedIn hosted in November 2016.

We believe Gradle Composite Builds is an example of a disruptive change LinkedIn can make by partnering with open source companies to improve developer productivity. The added benefit here is not only for those of us at LinkedIn, but also for all organizations that use Gradle. Never before has it been so easy to setup the IDE to work with code from separate code bases!