How Experimentation Helped LinkedIn Improve Email Communication
October 28, 2015
This is the second in a two-part series about how LinkedIn’s different business units use A/B testing and XLNT, our A/B testing software platform, to build better products for users. On Wednesday, we discussed how experimentation helped LinkedIn improve our Premium services. Today, we’ll discuss the benefits of experimentation on how we communicate with our members over email and through notifications.
Email messages and mobile push notifications are the two primary offline channels (not requiring direct interaction with LinkedIn apps or web) employed by LinkedIn to communicate with members. This communication includes: member-to-member invitations, periodic email digests (e.g., LinkedIn Groups digest, “What You Missed”, network updates, etc.), endorsements, and job recommendations, among dozens of other types of messages.
Email and push notification channels are of special interest because of their unique capability to reach members and help them connect with new opportunities, without requiring them to be actively logged into one of the LinkedIn apps (or web-page). On the surface, emails and notifications can bring great value to our members; however, their indiscriminate use can greatly increase the risk of requesting unwanted interaction and creating a negative user experience overall. How can we make sure we do the best for our members?
LinkedIn is constantly looking for better approaches to maximize the benefit to our members. This often requires choosing an optimal trade-off between the benefits (e.g., the message was read and there was a useful member interaction) and the disadvantages or risks (e.g., the message was ignored or marked as spam). One of the tools that LinkedIn uses to achieve this consists of large-scale constrained optimization, which helps finding optimal trade-offs in a largely automated manner. However, this by itself is insufficient. LinkedIn’s communication eco-system encompasses a large number of products, and changes in the underlying email distribution have complex effects in the system that could be challenging to predict offline. In order to understand the effect of these approaches LinkedIn relies heavily on running large-scale online experiments. Experimentation is critical at various stages:
When a new method for optimizing email distribution is designed, this is launched internally first (to company employees) and then to a very small random subset of users. This is easily done using XLNT, LinkedIn’s central experimentation platform, which keeps track of what experiments and variants are running for all of LinkedIn. It also allows for customizing the launch by targeting a specific subset of users (e.g., active users or users in a particular geographical region). Any change in the definition of these experiments is also done in a similarly efficient manner.
The impact of these experiments can be complex and distributed across several areas. Proper measurement of this impact would not be practical without reliance on our experimentation platform. XLNT provides a set of engagement metrics, including email sends and clicks, downstream page-views and sessions, and importantly, member complaints/unsubscribes. The insights provided by the experimentation platform do not stop here since the large collection of metrics supported by XLNT let us further measure the impact across many different products and areas. For example, we observed that a send volume reduction in some email types have a statistically significant increase in user engagement with other emails.
Proper understanding of the experiment’s impact requires adequate statistical analyses. XLNT automatically computes useful statistical quantities, including statistical significance for any metric change, the power of the experiment (i.e., will we be able to observe the effect we expect given the sample size and the metric of interest), and the expected site-wide effect if the size of the targeted population were to increase. In addition, by automatically calculating sample-size ratios for different experiment variants, cross-experiment interactions can be uncovered. In particular, this allowed us to discover unexpected interactions between two experiments that were both making send vs. no-send decisions, for a particular member segment.
This combination of optimization and experimentation has been highly effective. So far this year, it has allowed LinkedIn to send 40 percent less email and reduce email complaints by half, with a positive member response. In addition, this has led to an increase in email engagement rates. This constitutes an important step toward our goal of communicating just what matters to our members. In the future, we will share more details about the optimization technology that helped LinkedIn getting this done.