The Science of Quality Growth
June 1, 2017
Co-authors: Ken Soong, Liz Li, Yang Zhou, and Xin Fu
Growth hacking is a common buzzword among startups and mature companies alike. At LinkedIn, we believe growth is about accelerating the realization of our company’s vision to create economic opportunity for every member of the global workforce. Growth is not about moving metrics using anecdotes and short-term tactics. A healthy and sustainable growth plan should go hand-in-hand with great products that ensure members continue to receive increasing value. When strategized correctly, growth can generate long-term sustainable boosts to engagement and retention, which ultimately leads to revenue and profits. When strategized incorrectly, growth becomes a game of chasing numbers, which often strays from the company focus and may actually detract from business.
At LinkedIn, we’re continuously tuning our growth engine and we’ve experienced first-hand the entire spectrum of outcomes. We’ve conducted thousands of growth experiments, analyzed nearly every aspect of our growth ecosystem, and developed the metrics to accurately capture the value we bring to our members. This blog post documents these practices in hopes that you’ll be able to leverage some of our experiences to optimize your own.
What ultimately differentiates between good and bad growth is how you measure the success of an initiative. For each LinkedIn product, we have one primary success metric, which we call our “true-north metric.” Running A/B tests and making decisions based on the movement of a predefined true-north metric ensures that all decisions push the product in the correct direction. Good examples of true-north metrics for growth are measures like long-term engagement or member retention. On the other hand, measures like pageviews or clicks are poor true-north metrics and are often referred to as vanity metrics—so named because they feel good to talk about in the moment, but do not imply product success and are not indicators of user value.
For a true-north metric like long-term engagement, we might relate it back to a desired outcome, such as “active members after six months.” This definition of the true-north metric has the advantages of being easy to understand and easy to compute. It works beautifully to understand the historical performance of your business and might even help you infer past initiatives that were especially effective. The major problem with this definition is that there is necessarily a six-month lag between kicking off an initiative and measuring its success. This delay between testing and decision is too long, and we should instead consider an earlier indicator of success.
Design your true-north metric
Given the difficulty of using the desired outcome as a simple true-north metric, we look instead for a custom metric that fulfills the same role without the drawbacks—in this case, the long delay. An ideal custom metric would not only be highly correlated with the desired outcome, but would actually be causal for the outcome.
At LinkedIn, we’ve invented a true-north metric called Quality Signup. Unlike net signup, which tracks all users who complete our registration form, Quality Signup tracks the number of new members who have taken steps to establish their identity and grow their network within their first week as a member. Specifically, we track new members who have listed an occupation, made their first connection, and are reachable by other members. We believe these are the basic requirements for any member to start receiving value on LinkedIn. This metric was formed through a combination of machine learning, data analyses, and discussion with the product team, and has been carefully crafted to be causal for long-term engagement while still being easy to explain and understand. When our product teams test new growth initiatives today, we only consider them a success when they boost Quality Signups. For example, an initiative that doubles the number of new signups without increasing Quality Signups is not considered a success for us.
Analyses and A/B testing shows that increasing our Quality Signup rate causes an increase in our long-term engagement rate.
The concept of a custom true-north metric is not unique to LinkedIn. In fact, many other companies have similar measures that serve as their true-north metric for growth. For example, Facebook is known to track users who connect with 10 friends in seven days. Pinterest tracks Weekly Active Repinners. And Slack tracks teams who have sent 2,000 messages. For the remainder of this post, we’ll go through the general steps and considerations that you should think about when creating your own custom success metrics. Although this isn’t a foolproof method to solve all your growth problems, it is a great way to start moving in the right direction.
Step 1: Start with value propositions
The first step in creating a custom true-north metric is to understand how your product aligns with the value propositions your business provides. Since metrics serve to measure and quantify a product against an existing goal, they are ineffective without strong alignment to the value propositions. In Step 3, you’ll be using machine learning techniques to objectively quantify this relationship, but it’s important to begin with this solid conceptual understanding.
For Quality Signup: We thought about how member acquisition as a product specifically aligned with LinkedIn’s three core member value propositions:
Get the right job
Build meaningful relationships
Step 2: Data collection, label, and features
Once you have a good understanding of how your product aligns with your value propositions, you’ll want to quantify that with data. To start, you need to comb through historical logs to gather samples of what you’re trying to measure. These could be samples of: member actions, contract conversions, file uploads, etc. You then separate positive outcomes and negative outcomes, which will later serve as the label for our machine learning model. Choosing a label that accurately captures your business is probably the most difficult and influential decision in this entire process. When in doubt, you should always refer back to the mantra of growth and ask yourself: “Am I measuring long-term and sustainable value?”
For Quality Signup: We gathered all new members from a six-month cohort as samples. We classified new members who were still active six months after registration as positive outcomes, and those who were not as negative outcomes. By using “active” as the label, we made the assumption that members who were active were the ones who were receiving sufficient value from LinkedIn.
Along with the labeled data, you want to gather all the features of your product that might be useful for predicting whether a sample is a positive outcome or a negative outcome. A good starting point for picking features is to refer back to Step 1 and think about how your product aligns with the business goal. You should include all features that help drive your business value, along with any feature that might only be indirectly related. Each feature should also be measurable within a reasonable time frame. It’s hard to go wrong here and when in doubt, it’s always better to over-gather than to under-gather.
Step 3: Obligatory machine learning step
Once you have your features ready, it’s time to make sense of how they relate to the label in your dataset. Ultimately, you’re trying to decipher the arrangement of features that is most likely to lead to a positive outcome. In the case where you have a large number of samples, you’ll want to take advantage of existing machine learning techniques to simplify this task. Most classification methods will work here, but we usually prefer to start with the simple and powerful logistic regression with L1 regularization. The beauty of L1 regularization is that it prunes highly correlated features, as well as irrelevant features.
After training your machine learning model, you’ll want to verify the model’s performance using standard validation techniques. If the precision-recall of the model meets your standards, then we can wrap up this step and move on. Otherwise, we can continue to add features to your dataset or try more complex nonlinear models until you’re satisfied with the performance.
For Quality Signup: We began with an extensive list with dozens of features for each member in our sample. After removing all the unnecessary features that were not good indicators of long-term engagement, our final metric contained only a few of the most relevant features without loss in performance.
Step 4: Making your metric actionable (drive product strategy)
A metric purely based on a machine learning model is great for measuring long-term value, but it might not be very effective for driving growth. We’ve learned from our experience that in order to drive growth, a metric has to be easy to grasp and easy to rally behind. The challenge is finding the correct balance between the precision-recall of the model and the actionability of a simplified recipe. In most cases, you should be willing to sacrifice a moderate amount of predictive power for simplicity.
Take for example the case of Facebook’s metric: 10 friends in seven days. This is a great growth-driving metric because it perfectly fits the conditions—it’s easy to grasp and easy to rally behind. However, it should be obvious that there isn’t a sweet spot at 10 connections in seven days. A member who only has nine connections on Day 7 and makes two more on Day 8, would not be considered a success by Facebook’s metric, but might be even more valuable overall. In reality, the number of connections and the amount of time is a spectrum, and the most accurate model would take both of these factors as continuous variables. Facebook is making the trade-off between predictive power and simplicity.
In Step 3, we talked about removing spurious and correlated features. At the end of this process, if your metric still contains too many features to easily explain, then it’s time to start removing features manually. The simplest way to approach this is to stack rank the importance (predictive quality) of your features and repeat the machine learning process while gradually removing the lowest ranking features. Continue to repeat this process until you’re satisfied with the number of features or your model performance is below your requirements.
Finally, to really simplify your metric into a recipe that can be casually thrown around at the office, you’ll need to assign concrete numbers to each of your features. You can run your machine learning model with a single feature at a time to get an idea of where to begin and then test the precision-recall of various recipes. In the end, you’ll usually want to round your recipe to something that’s easy to remember.
Step 5: Validating your metric in its ecosystem
The steps up to this point have guided you towards designing a true-north metric that accurately measures long-term sustainable growth, while staying simple and actionable. At the very least, this metric is correlated with growth, but it’s also very likely to explain the drivers of growth. Distinguishing between correlation and causation using historical data is a recurring dilemma and is difficult to answer, even with the most sophisticated statistical methods. The only definitive way to prove that your metric is causal for growth is by passing it through the fires of A/B testing. A/B testing your metric not only proves its causality, but it also opens up a whole new set of strategies for driving your business through the insights surfaced by the definition of your true-north metric.
For Quality Signup: The new member acquisition team has focused its efforts on driving Quality Signups, even at the expense of some near-term wins. By focusing on the fact that Quality Signup was causal to long-term engagement, we were able to push through some initially alarming results and find material wins for long-term sustainable growth. For example, when we re-designed the LinkedIn guest homepage, we saw a decrease in signups, but no change in Quality Signups. We dug deeper and discovered that members with existing accounts were more likely to login with the new design and we were actually reducing the number of unintentional duplicate signups! Without this metric, we might have terminated our experiment prematurely.
Think about how the features in your metric integrate into your product ecosystem and diagram how the product moves your true-north metric in a funnel. Once you understand this funnel, create initiatives that can directly drive your true-north metric and verify that they also lead to long-term sustainable growth. In practice, there’s usually a long delay between seeing an impact on your true-north metric and seeing its effect on long-term growth, but that’s expected by the nature of these two different measurements. Once you’ve shown the causal connection between your true-north metric and growth, you can focus efforts on driving your true-north metric while remaining confident that it’ll lead to real growth in the future.