How LinkedIn automates cherry-picking commits to improve developer productivity

Our developers at LinkedIn are constantly exploring ways to enhance and strengthen our platform, aiming to provide our members and customers with the greatest possible access to knowledge and connections. With approximately 15,000 code repositories, our developers work tirelessly to make thousands of code changes each day, improving functionality and resolving any issues that may arise. Most of these repositories practice trunk-based development, which means developers push code changes frequently to the trunk (main branch) and avoid any other long-lived branches. Code changes are peer reviewed using a pull request (PR) workflow on GitHub and merged into the main branch.

While most repositories (e.g., web services) practice continuous delivery solely from trunk (or main branch), those (e.g., core libraries) that need to stabilize their release or release on a fixed schedule use release branches. Sometimes when developers make some code changes in the main branch, they want to apply those changes to release branches as well so that those branches also get the same benefit. In this case, they use the ‘git cherry-pick’ command to manually apply changes across branches. This is a really mundane and time-consuming task for any developer.

To automate this task, we developed the Automated Cherry-pick GitHub app which significantly reduces efforts of developers working at LinkedIn. Now cherry-picking commits only takes one developer a few seconds as opposed to multiple developers spending 10+ minutes without this automation, thereby reducing the time that developers need to spend on cherry-picking commits by around 99.6% This newfound efficiency means now developers can spend more time and focus on delivering the insights and information that our members and customers value most.

Figure 1: Manual Cherry-pick vs Automated Cherry-pick: User Actions

The Tale of Two Options

Imagine two developers, Alice and Bob, collaborate on a GitHub repository with two protected branches: trunk and maintenance_v4. Trunk is the development branch for the current major version (v5). The maintenance_v4 branch is used to make critical bug fixes and security upgrades to the previous major version (v4), so consumers who are still on v4 can pull in critical fixes without having to simultaneously do the major upgrade.

Manual

Assume Alice opens a pull request to fix a bug in the trunk. Bob approves the fix and suggests cherry-picking it to maintenance_v4 since the bug fix is security related. Alice then kicks off a long list of mundane tasks. She checks out the maintenance_v4 branch locally, cherry-picks the bug fix to it using git, runs the tests locally, creates a PR targeting maintenance_v4, and asks Bob for another code review before merging the PR. Now imagine the repository had more than one active release branch (e.g., release_v5.1, a release branch to stabilize the future v5.1 release). Alice needs to manually cherry-pick this bug fix to those branches as well and Bob needs to approve them again (i.e., more manual work)!

This manual process is complicated, ad-hoc, and time-consuming. As a result, these steps may be neglected or skipped by Alice and Bob when the effort outweighs the gains (e.g., for a low impact bug fix), which can lead to known bugs that are fixed on the trunk to resurface in future releases from other branches. 

Automatic

Automated Cherry-pick automates this process and makes it a standard part of the code review cycle baked into the system. For instance, in the above example, Alice can simply add cherry-pick labels to her PR for each target release branch. Our automation then cherry-picks the fix for her by creating a PR for each of those branches, and merging it automatically once the tests pass! Alice is happy because she no longer needs to spend ~10 minutes per cherry-pick, and Bob is happy because he no longer needs to put his colleague through tedious work.

How does Automated Cherry-pick work?

Automated Cherry-pick is a GitHub app that provides developers a way to cherry-pick changes in a pull request to other protected branches. It’s not just cherry-picking changes from one branch to another, it’s an end-to-end cherry-pick workflow embedded in the code review process at LinkedIn to provide a seamless developer experience.

Figure 2: Automated Cherry-pick: End-to-end Workflow

Once developers create a pull request targeting a protected branch, they can specify other protected branches (e.g., release) using PR labels in the form ‘cherry-pick to <branch-name>’ as shown below (Figure 3) to cherry-pick the change proposed by the PR into them. Immediately after adding PR labels, we validate if all labels are valid – meaning that the branch mentioned in the PR label should exist and it should be a protected branch. Invalid labels are automatically removed, and a corresponding comment is posted to the PR. This early feedback is helpful to developers so that they can provide valid cherry-pick labels.

Figure 3: Automated Cherry-pick Labels for LI-AGP PR

Once the PR is merged, we launch the Automated Cherry-pick process. The Automated Cherry-pick process creates new PRs to cherry-pick the change to the branches specified in the original PR labels. For each branch, if the cherry-pick is successful, a link to the cherry-pick PR is posted as a comment on the original PR. If the cherry-pick is unsuccessful (e.g., because of merge conflict), the corresponding failure reason is posted as a comment on the original PR so that users can fix it.

Figure 4: Comment on LI-AGP PR - Cherry-pick Initiated

Those newly created cherry-pick PRs have the same description as original PR and information about the cherry-picked commit as seen below in Figure 6. Cherry-pick PRs are treated in the same way as regular PRs which means they go through checks like PR description check, code sanity check, unit testing, compatibility testing, and so on. If the person who added labels is the owner of the repository, the cherry-pick PR is auto-merged after passing all checks. Otherwise, approval is requested from the owner of the repository and cherry-pick PR is merged after owner approval.

Figure 5: Comment on LI-AGP PR - Cherry-pick Successful

Figure 6: LI-AGP Cherry-pick PR

Primary Use Case at LinkedIn

Several teams at LinkedIn use Automated Cherry-pick, but one of the primary use cases we have is LinkedIn’s customization around Google’s Android Gradle Plugin (AGP), called LI-AGP, whose release cadence and versioning policy is mapped to AGP.

AGP goes through an alpha, beta, and Release Candidate (RC) process for every new version, with active development happening on the alpha channel. Once a release moves to beta, new feature development is typically frozen and only bug fixes are included in new builds. When a beta moves into RC, the branch is frozen and only the most critical fixes are picked up. At any given time, there are typically three versions of AGP available: Stable, Beta, and Alpha.

LI-AGP’s main branch always targets the latest alpha release of AGP. It allows us to test AGP releases early and give Google feedback in the alpha stages, while still using stable versions for our apps. When AGP releases a new major alpha version, we update LI-AGP's main branch to the new version, then create a release branch for the previous version. The release branches are updated as AGP versions move from alpha to beta, and so on. So, whenever someone fixes a bug in the main branch, they backport it to release branches using Automated Cherry-pick.

Conclusion

Automated Cherry-pick improved developers’ productivity by reducing one of the tedious tasks they need to do in day-to-day work. As of now, we have enabled this automation for 3,000 repositories and we plan to enable it to all our 15,000 repositories in the future. In the three months, only LI-AGP has used cherry-pick automation on 28 PRs and created 54 cherry-pick PRs, as changes in one PR can be cherry-picked to multiple release branches resulting in multiple cherry-pick PRs. Assuming the average time required for one manual cherry-pick is around 10 minutes, the Automated Cherry-pick saved about nine hours of developers’ time for the LI-AGP use case. In the future, LinkedIn is planning on moving to GitHub Actions, at which point we will be exploring how to refactor the cherry-pick app into a GitHub Actions workflow.

Acknowledgements

Individuals from different teams supported us in this project. I would like to thank Nima Dini for coming up with the idea and design of this project. Thank you Kedong Shao and his mentor Niket Parikh for laying a strong foundation for this project during your internship and providing it a good shape. I would like to thank my manager Brian Beck, and colleague Zach Yang for providing support during this project. Thank you to Drew Hannay and Scott Jasso from the Android team for being early adopters of this feature and providing nice feedback. Last but not the least, thanks to Jinzheng Sha, Prince Valluri, and Stephen Yeung from the Code Collaboration Platform team for helping us with GitHub integrations while working on this project.