You’re Only as Good as Your Lieutenants

This post is part of the series “Every Day Is Monday in Operations.” Throughout this series we discuss our challenges, share our war stories, and walk through the learnings we’ve gained as Operations leaders. Today’s stories “Priority Zero” and “Scale or Fail” come from David Henke and Benjamin Purgason, respectively. You can read the introduction and find links to the rest of the series here.

It doesn’t matter if you are a new engineer fresh out of boot camp or the CEO of a Fortune 500 company: you only have 24 hours in a day, just like everyone else. How you allocate your time is up to you, but no matter how efficient you are, you still have an upper bound at 24 hours.

To have an impact that is bigger than a mere 24 hours, you have two options: scale linearly by automating your work, or scale exponentially by inspiring others to work towards a shared objective. The latter is the definition of leadership and the key to staying ahead of the curve in a world where every day is Monday in operations.

Priority zero

A few years ago, I was invited to speak in front of a group of police lieutenants. This group contained members from various police forces across California, from Oakland and San Francisco to Los Angeles and San Diego. My topic? Leadership.

As I sat down to prepare my speech, I wracked my brain. How could I get a group of public servants that uphold law and order for the state of California interested in something that someone from the private sector and the internet had to say?

I told a story that started with my brother and I going to a Los Angeles Lakers game. The scores don’t particularly matter because I didn’t get to watch the whole game, and neither did he. We had barely sat down before I received the call: priority zero—the site was down and we were not serving any pages at all. Shortly afterwards my brother, the superintendent of the Montebello Unified School District, received a call: priority zero—there was a shooting at one of his schools.

After I recounted this story, I turned to my audience: what is your priority zero? They responded “officer down.” I know that I do not have the worst priority zero, but regardless of the type and severity, how we respond to priority zero events is all about leadership. How do you protect your team as best you can from the unthinkable? You are only as good as your lieutenants.

When I retired from LinkedIn, they threw me a party, hosted by the top 22 engineering leaders in the technology organization. In between talking with the people in attendance I looked around the room and something struck me. I realized that of the 22 engineering leaders scattered throughout the crowd, five of them predated me at LinkedIn. The other 17 leaders I had hired directly, including my own successor. Why did I notice this, of all things, right at the end of my time at LinkedIn?

When I first started at LinkedIn, complete site outages happened far too often. What’s worse, the culture was built on a one-way flow of information—the product team would get an idea and instruct the engineers to implement it. The engineers would implement an idea as quickly as possible, then instruct operations to support it. The operations team would support it by fighting fires all day, every day. The security team? It was way too small.

My priority zero: break the cycle and change the culture so that this situation could never happen again. I went on a hiring spree, and within six months, I had brought in experts in data science, site reliability, data center architecture, networking, scale computing, and information security.

I didn’t hire the “best technical engineer,” I hired the best engineers I could find that understood collaboration. These were all great engineers who understood what it takes to build a team and understood the need for teams to work together.

As the party began to wind down I knew that I was leaving LinkedIn in good hands.

Learning from “Priority zero”

As we’ve said before, technology is easy to change, but culture is not. The cultural shift we implemented was the greatest contribution we could give to LinkedIn. Years of effort yielded incredible results: at LinkedIn, talent is our number one priority, and our people come first.

Whether you are thinking about your company’s well being, your team’s, or your own, priority zero should be the long-term development of your lieutenants. These leaders you help develop will ultimately take over the roles and responsibilities you have today, freeing you up to take on an expanded role. Your company wins through the improvement of its talent pool and improved ability to provide continuity through unusual or difficult circumstances.

Scale or fail

My first job when I made the switch from being an individual contributor to a people manager was to build out the Tools SRE team. I was confident that I understood the challenge ahead of me: there were applications that predated us by years, operational challenges, and a big team that we’d need to partner with. That meant balancing incident management, making improvements to older applications, and being a part of the design of new applications.

At first, I approached my task in the same way I had approached all of my prior technical projects. I gathered my new teammates around, divided up the work, and everyone began working. I would make the decisions, lead the incident response, and be a part of just about every technical conversation. It wasn’t long before I learned that every day is Monday in operations and that my strategy was not going to work.

The rate of change was high and this caused me to become the obvious bottleneck. I was caught context switching: either leading incident response efforts, sitting in meetings, or making decisions without enough information. I was overloaded and my team was paying the price; they couldn’t make meaningful progress because I was in the critical path for too many decisions, and they were growing frustrated.

One weekend, I was thinking about how things were going and trying to figure out what I needed to do. I mused over the things we could do if I had a team full of “engineers like me.” I didn’t need more me, I needed a team of engineers, each better than me.

I realized that I needed to spend my time doing what only I could do: helping my team grow and exceed my own engineering capabilities. On Monday I called together my team and explained to them that I wanted to try something new. I set our team’s mission, vision, culture, and purpose.

I explained my expectations and then I offered each of them the opportunity to step up and claim responsibility for an idea we needed to execute on. They were challenged to make decisions, take intelligent risks, help LinkedIn win, and to do all of that in keeping with our newly established culture. My role, I explained, was to help them grow, provide guidance, and remove roadblocks they encountered.

At first the going was rough, but over time, the founders of Tools SRE grew from a ragtag band of executors into a group of capable leaders. They embraced the culture, amplified it, and asked for greater challenges.

Today the team members are capable, take intelligent risks, and build trusted relationships easily with their partner team. They now work to raise the next generation of leaders.

Learning from “Scale or fail”

Every leader has a different set of starting circumstances, but one thing stays the same: the sooner you start spending your time on what really matters—your people—the better off you will be. You cannot go it alone. You will fail. Create a culture that empowers your team members, helps them grow, and that they want to be a part of. Eventually, you’ll end up with a team that is greater than the sum of its parts.

Relationships matter. If your team trusts itself and you have a shared culture providing guidance in uncertain situations, you can move incredibly fast. You can deal with things that are impossible for a single engineer, no matter how good they are, to deal with. MTTR will be reduced, you can better uphold site up, and you have a chance to get ahead of Monday’s problems.