At LinkedIn, we strive to build products that our members love. To help make informed product decisions, we do a lot of A/B testing. Although it is a well understood process on the web, it presents a significant challenge on native iOS and Android apps because of their release cycles and singular shipped binary. In this post, we will give an overview of how we are able to modify the client side UI based on well-defined view based JSON responses without client iteration. We will also show how we use this ability along with the LinkedIn Experimentation infrastructure to do highly targeted context-aware A/B testing at scale through our phone server. Finally, we will provide a sneak-peek into how we use all the collected data to automatically generate sophisticated comparison graphs between different buckets to make decisions.
View based JSON
All of our mobile clients (Android, iPhone and Touch Web) render content based on structured view based JSON. This means that these clients have a set of predefined views for which data can be populated with JSON associated with that view type. The top slot in the user's stream is called the hero slot view. One of the possible JSON responses for the hero slot looks like the following:
The vType (view type) tells the client which view to render and all the other fields are directly translatable to the requirements of that view such as the background picture, heading text and detail text. Notice that the fields are based on the visual design of the hero slot and you can directly map them to the visual presentation of the hero slot. This is different from a resource based JSON in which the entity being sent down from the server is a news resource like the following. In this case, the resource is then translated by the client to something like the view based JSON above by extracting the required information from the resource.
How does view based JSON help?
- The client does not need to understand the complexities of the news resource and a client iteration is not required when the resource signature changes on the server.
- New resources can be formatted to the same view type to automatically show up on the client without requiring a client iteration. For example, influencer follow updates could be formatted in the same way as company follow updates.
- The server controls exactly what shows up on the client. The view rendered on the client is decoupled from the complexity of underlying data model.
- Sorting based on relevancy can be accomplished on the server. The client need not be involved in deciding which item shows up where.
- Bug fixes are easy! If a certain stream update is buggy on a certain version of the client on a certain OS, we can disable sending down that view type from the server and the client simply stops showing that particular stream update type.
- As a side-effect, it encourages consistency in the look and feel across the app, since the design dictates the view based JSON and one can reuse views for similar entities.
- Finally and most importantly, A/B testing and experimentation becomes as easy as sending down different view types for the same item to the client. Let's explore this further in the next section.
A/B testing with view based JSON
In the above example, the A/B test decides whether the top slot in the user's stream should encourage the user to add more contacts by importing her/his address book or show top news articles. While building the stream, the phone server queries LinkedIn's internal member segmentation and A/B testing platform called XLNT (pronounced Excellent), to decide which bucket to put the user in. XLNT targeting selectors like the user's number of LinkedIn connections are used to refine experiment targets in this case. This is combined with last viewed timestamps to make the decision for the current user. Since our clients perform rendering based on the view type, the decision process is entirely on the server. This process is described in detail in the next section. Finally, the clients, all three of them, receive one of the following view based JSON.
Notice that the client happily renders the different treatments of the top slot in the stream based on the view types and handles user actions based on the corresponding link type. There are again multiple advantages to this approach.
- We can be sure about the accuracy of our treatment buckets as none of the decision process is on the client.
- Tweaks to the experiments can be made on the server and new buckets added/removed without requiring client iterations.
- Highly targeted experimentation can be performed based on client request context like client version, OS version, OS name, carrier, locale, language etc. Let's explore how we achieve all this on the server in the next section.
LinkedIn Experimentation Platform
XLNT (pronounced Excellent) is an A/B testing platform at LinkedIn powering both mobile and desktop experimentation. We can use the platform to enable a feature for a specific subset of users based on their attributes: language, geo, date of enrollment into LinkedIn etc. We have hundreds of tests defined in XLNT for mobile experimentation. An experiment definition conceptually looks like the following:
In this experiment, 40% of Android users are shown two variations of a Call-To-Action (20% each) to endorse someone for a skill on their profile page. The three buckets for the experiment are A (action_promo), B (hero_promo) and no CTA (control). On the phone server, for each incoming request from any client, we receive the following contextual information about the client.
This information is used to query XLNT for all the experiments that are relevant to that request. The contextual information is passed along to the XLNT backend which then assigns an appropriate bucket to the client based on the experiment definition. Based on the assignment, the Node.js phone server formats data for the client by assigning it the appropriate view type for that treatment as shown in the following javascript code snippet. The client is completely oblivious to this bucketing.
To close the loop, we need to measure the performance of different buckets to make final product decisions and iterate on designs that work well. We normally want to optimize relevant metrics like click through rates and page views. This is achieved in the following way.
- The phone server queries the XLNT backend for an experiment as described above. At the same time, an event is sent to XLNT recording the bucket served to that client along with the contextual information about the client like OS name, client version etc.
- XLNT automatically associates all this data with metrics that we are trying to optimize like page views or click through rates.
- An XLNT dashboard for each experiment is generated that displays the effect on these metrics along with relevant statistical information allowing us to make data driven decisions.
- Deploying the chosen variation is now as simple as tweaking the XLNT experiment definition to assign 100% of the users to that bucket. No client iteration, no server deployment, instant gratification.
Interestingly, it's possible to drill down on these metric comparisons based on parameters like OS name. We have had instances where a certain variation works great for Android whereas a certain other variation works better for iOS. Once again, XLNT targeting selectors allow us to target users based on their device types and assign them different variations based on the results of the A/B test.
Conclusion
Our mobile A/B testing process has been highly successful in helping us make great product decisions based on real data from our users. Rendering based on view types on the client side allows us to dynamically modify client UI for any subset of the clients with the entire control on the server. The XLNT framework allows us to perform targeted experiments that can also leverage contextual information provided by the client. It also provides a sophisticated system to analyze the collected data in a visual manner and make decisions with ease based on relevant engagement/growth metrics. At LinkedIn mobile, we are still exploring numerous other ways to measure and improve user experience for our users on iOS, Android and Touch Web and we can't wait to share what we learn.
Try out our latest iPhone and Android apps and keep helping us shape your favorite LinkedIn apps.