LinkedIn just released completely redesigned versions of its iPhone and Android apps. As part of the redesign, we are introducing a new personalized navigation experience to help members explore new functionality and quickly get to the areas they access the most. In this post, I'll give you a brief idea of how we use LinkedIn's Hadoop infrastructure to power personalized navigation recommendations.
The Challenge
Real estate in a mobile UI is too limited and too valuable to be cluttered with navigation options. We needed to find a way to make the best use of the little space we have. Think of the various sections of an app as rooms in a really big house: we want to provide shortcuts to the rooms most accessed by residents of the house. In effect, the house molds itself according to the needs of the current resident and occasionally guides him/her to explore an uncharted room.
The navigation items can be divided into two broad categories: branches and leaves. Branches are the broad sections in the app which represent a product group like Companies or News. Leaves are the individual pages in the app like a particular company's page, a group's discussion list or a person's profile. The objective is to analyze a member's usage patterns to automatically populate their navigation area with the branches and leaves that would maximize engagement with the app.
For example, my personalized navigation area includes News, Connections and the Stanford University group:
Overall Pipeline
Here is a high level overview of the recommendation pipeline:
- We track each page view and action by a user and send the data to Kafka, our distributed messaging system.
- On a daily basis, we consume and store the Kafka events in HDFS.
- A weekly Hadoop workflow (described in the next section) consumes the HDFS data and generates per member recommendations that are stored in Voldemort, our key-value storage system.
- Our Node.js phone server queries the Voldemort store for recommendations and maps it to relevant navigation items to be displayed in the UI.
- Taps on the recommended items and corresponding product usage feeds back into the system to guide the next weekly run and results in evolution of recommendations with changing usage patterns.
Hadoop Workflow
Once per week, we run Hadoop jobs to crunch all the usage data and come up with navigation recommendations:
The first step is to take the raw action and page view data in HDFS and process it to figure out which features each member is using and how often:
- We use a mapping table to convert each action and page view to a product branch or leaf.
- For each member and each branch/leaf, we count (a) the number of weeks the member visited the branch/leaf at least once and (b) the number of days they accessed the branch or leaf in the last week.
- When counting, we use a scaling factor to control the influence of events. For example, we scale mobile events up and desktop events down so that the actions you take on mobile have a higher impact on the navigation menu you end up seeing.
For example, here are what the counts might look like for a member 123:
| m_jobs_view | Jobs | Branch | 7 | 2 |
| d_companies_action_tap | Companies | Branch | 1 | 3 |
| m_group_456_view | Group 456 | Leaf | 2 | 6 |
We can use these counts to decide which items to show in the navigation drawer:
- We start with a very small default set of navigation items based on the historical usage of all members.
- Next, we calculate a threshold to decide how many times a member had to access a branch/leaf for it to make it to the recommended list of items. We aim for a threshold that strikes a good balance between the number of automatic recommendations and user choice to later add more items through the Add Shortcuts menu item.
- Items that pass the threshold are marked in green in the table above. We sort them by usage count, with a higher preference for continued usage over multiple weeks instead of spiky usage in one particularly active week.
- We cap very active members to 3 recommendations; inactive members get the default set unchanged.
After all this processing, here are the recommendations we would store in Voldemort for member 123:
| 123 | jobs, leaf_group_456 |
Conclusion
Approximately 56% of our users will see at least one recommended navigation item and roughly 33% will see more than two recommended items. As expected, we saw Who's Viewed Your Profile being recommended to 55% of our users; People You May Know and Groups followed closely behind. As this project evolves, we are planning to use collaborative filtering techniques to generate recommendations for new or inactive users who currently only see the default set of navigation items. We also plan to use profile characteristics like job-seeker or recruiter to guide the selections.
We believe this system makes it easier for our users to navigate the application and have the UI adapt to their individual preferences. Of course, there is still a lot to improve. We'll be closely monitoring feedback from users and will analyze the impact on user behavior through A/B testing. We have also introduced relevance models in other parts of our app, such as the update stream. We hope you enjoy using this app as much as we enjoyed making it!
Try out the new iPhone and Android apps and let us know what you think!
Acknowledgements
Special thanks to Leah McGuire.