Editor's note: Interested in performance engineering topics? Consider attending the Performance Engineering Meetup on Wednesday, March 21 at 5:30pm in Sunnyvale (RSVP here).
The mission of the Performance team at Linkedin is to deliver a delightful experience to every member of the global workforce with applications that feel instantaneous. In order to achieve this, the team develops and maintains tools and frameworks to measure, and analyze every member interaction and works with teams across engineering on critical optimization projects. Some of these projects are classified as “horizontal initiatives” within LinkedIn.
Horizontal initiatives at LinkedIn are distributed engineering projects where multiple engineering and/or operations teams are required to execute predefined tasks that are not part of their development roadmaps. Cross-organizational projects that require contributions from multiple engineering teams qualify as horizontal projects if they are centrally executed and consist of tasks to be executed on behalf of multiple other engineering teams and require their buy-in and/or input. Horizontal initiatives often remove tech debt, upgrade our stack, improve sitespeed, or do something else to make everyone's life easier within LinkedIn Engineering. Notable examples of horizontal initiatives at LinkedIn range from optimizing page load times to upgrading our federated search stack to killing LEO, LinkedIn’s original monolithic application. Having a special classification for these cross-team projects has several advantages. For example, the elevated attention paid to these projects combined with the network effect of having several different engineering orgs involved in steering the horizontal project means that it can be easier to get a team to change the prioritization of their work when it is blocking the horizontal initiative. The institutional knowledge embodied in the horizontal team also means that we are better able to predict the likelihood of success for different cross-organization projects.
Principles for scaling decision-making across teams
One of the critical requirements for the successful execution of a horizontal initiative is clear alignment of participating teams with regard to expectations, decision-making processes, and organizational communications. The Performance team achieves alignment with teams across the organization by adhering to the following principles:
Developing a common understanding of the impact of proposed projects
Arriving at mutually agreed-upon priorities
Getting agreement on the time-to-market/timeline
Using the RAPID framework to identify the roles of collaborating teams based on skill sets, context, domain expertise, and resource constraints.
This framework is derived from the process and management practices recommended by Fred Kofman, VP of Leadership and Organizational Development at LinkedIn.
Develop a common understanding
One of the difficult challenges for any organization is to make decisions with incomplete and decentralized information about costs and benefits. Each team has data about its service, system, or infrastructure, and partial knowledge about how changes to their service or infrastructure can affect the performance, scalability, and availability of the entire system/site. But no one team has the global context or full knowledge about how their service can affect the entire system.
Developing a common understanding of the problem, that is, the benefits and impact of proposed projects for system performance, stability, and efficiency, and their contributions to business metrics, will help teams to communicate and collaborate effectively.
Arrive at a mutually agreed-upon priority
The business environment in the digital economy is dynamic, meaning business requirements and priorities of engineering projects are constantly shifting. In any organization, especially a large organization like LinkedIn, management and teams have to navigate competing priorities and commitments to serve the needs of their customers and stay ahead of the competition. In this environment, it is common for teams to have different or contradicting perceptions of priorities. Performance optimization projects can often compete with, and consume time and resources from, other engineering activities (e.g. feature development and upgrades). In such situations, it is the responsibility of the team proposing the optimization (i.e, the Performance team) to provide context regarding the benefits of the proposed project and its cost and risks with supporting data. This helps engineering teams to agree on the priorities of the project, based on the evaluation of the costs and benefits.
Clean escalation
In situation where it is difficult to arrive at a mutually agreed-upon priority due to a) lack of data, b) roadmap/timeline commitments, or c) failure to arrive at a common understanding, clean escalation should be employed. Clean escalation could involve senior engineers, management, or senior executives in the discussion to help integrate the information, bring a more systemic perspective, and make a judgment call if necessary. You can learn more about clean escalation here.
Get agreement on time-to-market
Once teams agree on the impact, cost, and priority of a project, it is important to communicate clear expectations regarding the timeline or time-to-market requirements and relevant justifications. The collaborating teams identify dependencies and get commitments on timelines/ETAs based on the product roadmap, business requirements, and resource constraints.
The RAPID framework
According to the Harvard Business Review, effective decision making in projects that span across functional teams remains a constant challenge for organizations of all sizes. For instance, a team that thinks it’s more efficient to make a decision without consulting other functions may wind up missing out on relevant input or being overruled by another team that believes—rightly or wrongly—it should have been included in the process. Many of the most important cross-functional decisions are, by their very nature, the most difficult to orchestrate, and that can string out the process and lead to misunderstandings between teams and costly indecisions.
The RAPID framework
Based on our experience, the most frequent cause of misalignment is lack of clarity of roles and responsibilities among the collaborating teams. To address this issue, it is important to discuss and decide on roles and responsibilities among collaborating teams during the initial discussions of a project. R.A.P.I.D is a highly effective framework and is easily adaptable to different situations. It untangles the decision-making process—existing or upcoming—by identifying all of the various roles and activities that must occur for the successful execution of a project. For a detailed description of the RAPID framework, please refer to RAPID Essentials.
The decision framework in action
Projects that require cross-team effort are expensive for the company and disruptive for the team, so it is important to have clear selection criteria that ensures high ROI for the organization from such projects. Projects at LinkedIn qualify as horizontal initiatives based on the following selection criteria:
- Importance - The project has a severe and broad impact on the company with active support from senior leadership.
- Urgency - The project has a hard deadline or interdependencies required by the senior leadership with severe consequences for missing them.
- Quality - The team has completed a proof-of-concept with the partner team validating the technical solution, the level of the effort, and the implementation plan. They have also completed all the prerequisite work and have created thorough and well-written instructions for the partner teams.
- ROI - The team has demonstrated that the project will save LinkedIn millions of dollars or will prevent a legal or monetary threat.
- Horizontal - The work will require substantial involvement across the entire company impacting multiple organizations.
For example, let’s walk through a recent horizontal initiative that used the framework described above. The Performance team proposed the use of service workers to help LinkedIn improve user experience, especially with the adoption of single page applications architecture. Since a service worker controls its own cache (independent of other websites), we could leverage this cache to serve an older copy of our application (HTML/JS/CSS) while we fetched the most up-to-date version in the background. This should have improved boot up time for repeat visits because users would not need to download the full application on each visit.
However, the Performance team does not own the presentation infrastructure at LinkedIn, and the implementation of the service workers needed the expertise and support of the Presentation Infrastructure (PI) team. The Performance team worked with the PI team by first proposing the idea and developing a common understanding of the problem and the benefits. We presented a design proposal and reviewed it with the PI team to get their feedback. We then gathered performance data from our initial prototype, which was helpful in establishing the impact and agreeing on the priority of the project. Finally, we used the RAPID framework to determine the role of each team in this project. Based on the expertise of the teams and the ownership of infrastructure/code, we decided the engineering leader of the PI team would be the Decider, or “D,” and the Performance team would be the Recommended, or “R,” and would work with the PI team to implement the optimizations. That way, both the Performance team and the PI team would be Performers, or “P.”
Conclusion
Much like tech debt, engineering projects can sometimes suffer due to “communication debt” or “culture debt.” The decision framework described in this post has helped the performance engineering team scale with the growth of LinkedIn and its engineering organization. If you are leading a project or initiative that spans across the organization, this framework will be helpful. It will help establish accountability to the right teams and provide greater clarity around ownership and decisions. This framework also helped foster better teamwork between the teams and reduced friction and delays in the project.
While the RAPID model may not be the one that your team decides to follow, having a process that develops a common understanding of the impact of proposed projects, arrives at mutually agreed-upon priorities and timelines, and establishes clear roles and responsibilities for all teams involved is key.
Acknowledgments
I would like to thank the many people involved in guiding horizontal initiatives at LinkedIn, but especially Brandon Duncan, Sarah Clatterbuck, Manish Dubey, Ritesh Maheshwari, and Michael Butkiewicz.