Communities AI: Building communities around interests on LinkedIn
June 24, 2019
At LinkedIn, our mission is to connect the world’s professionals to make them more productive and successful. Part of the way we achieve this goal is by providing a platform where communities connected by common interests or shared experiences can form. Several features—for instance, Groups—help foster the growth of these communities. However, an integral tool for connecting members to each of their various communities is the LinkedIn feed. The feed provides a place for our members to discover and join conversations happening among their connections or within their groups.
As we’ve discussed previously on this blog, artificial intelligence (AI) plays an important role in sifting through the myriad updates and posts available to serve to a member when they view their feed. These efforts are managed by the Feed AI team within LinkedIn. There are many different parameters to optimize for, however, within the larger umbrella of feed relevance. To assist these efforts, we have a dedicated Communities AI team focused on using AI to serve members relevant content that sparks interest and conversation within the communities they’re part of.
In this post, we give an overview of the techniques that the Communities AI team uses to help form communities and conversations around common areas of interest. These techniques include: follow recommendations, topic-based feeds, hashtag suggestions, and typeaheads.
Fostering active communities at LinkedIn can be broken down into the following components:
- Discover: Help members find new entities to follow that will connect them with communities that share their interests.
- Engage: Engage members in the conversations taking place in their communities by serving content from their areas of interest.
- Contribute: Help members effectively engage with the right communities when they create or share content.
These three aspects are all part of the same ecosystem and our goal is to build an AI platform that closes the loop between Discover, Engage, and Contribute.
There are many ways we can determine a member’s interests. However, we focus here on the explicit signal the member provides by following different entities (e.g., following companies, other members, influencers, hashtags, or joining groups). The vast majority of the follow connections that happen on LinkedIn are driven by the Follow Recommendations product (see Figure 1) that is developed by the Communities AI team.
Figure 1: Example follow recommendations for a member. This example shows hashtag and company recommendations.
The goal of the Follow Recommendations product is to present the member with follow recommendations that the member finds both relevant (i.e., increase the probability the member will follow the recommended entity) and engaging (i.e., the recommended entity produces content that the member finds relevant). Estimating relevance and engagement is where the bulk of the machine learning work happens. To compute these estimates, we rely on information (features) from the viewer (member), the entity to be followed (e.g., influencer, company, hashtag, group), and the interaction between the viewer and the entity (e.g., the number of times the member viewed the feed of a specific hashtag).
There are over 630 million members on LinkedIn. This presents a scaling challenge and a relevance challenge. The Follows Relevance data flow processes hundreds of terabytes of data and is the second largest at LinkedIn after the People You May Know flow. To understand how we managed this explosion of data, we refer the reader to the article Managing "Exploding" Big Data.
As soon as a member follows an entity, the content generated from that entity starts flowing to the member’s main feed and is ranked in conjunction with other content (e.g., posts from 1st degree connections).
The member can also go to specialized feeds for each entity that they follow, be it a hashtag, group, event, etc. We personalize these feeds by ranking more relevant content higher. For example, a post in the #AI feed that is posted by an influencer the member follows is more likely to be relevant than a post that is generated by another member.
The goal of feed ranking at LinkedIn is to help members discover the most relevant conversations that will aid them in becoming more productive and successful. Relevance is determined by our objective function which optimizes for three main components: The value to the member, the value to the member’s network (downstream effects), and value to the creator of the post. A diverse set of machine learning and experimentation techniques are used to estimate these three components and the combined effect of the three (e.g., see Spreading the Love in the LinkedIn Feed with Creator-Side Optimization).
Our Hashtag Suggestions and Typeahead (HST) product recommends hashtags that allow the member to effectively target their posts to the right communities. In addition to reducing the friction the member faces when trying to add hashtags to their post, the HST product allows us to consolidate content around areas of interests and prevent content fragmentation.
The objective of HST is to both increase the probability that a member will select relevant hashtags from the recommended list to add to their post and increase the relevant feedback the member will get on their post. Here we use a variety of natural language processing (NLP), deep learning, word embedding, and supervised learning techniques to recommend relevant hashtags. The HST product is shown in Figure 2 below.
Figure 2: A demonstration of the HST product in the LinkedIn Share Box. HST suggests hashtags while the member types their post.
Interactions among Discover, Engage, and Contribute
A member’s journey does not necessarily have to happen in the order given above. In addition, a member’s behavior in one aspect can be a valuable signal in another aspect. For example, if a member tags a post with a specific hashtag that the member does not follow, then the Follow Recommendations product can use this signal to determine the relevance of this hashtag to the member and recommend the hashtag to be followed.
The Communities AI’s vision is to use Artificial Intelligence (AI) to serve relevant content to members based on what they are interested in and help members engage with each other. We’ve explained the three aspects of our communities ecosystem (Discover, Engage, and Contribute). In addition, we’ve highlighted some of the technical challenges in each aspect.
There are more details that we intentionally skipped because they are out of the scope of this article. Stay tuned because we’ll likely discuss these details in future articles!
It takes a lot of talent and dedication to build the AI products that enable communities on LinkedIn. We thank all the members of the Communities AI team (Ankan Saha, David Golland, Brian Olson, Andrew Hatch, Suman Chakravartula, Mohammad Rafiee, Emilie De Longueau, Aubrey Gress, Ian Wood, Daniel Ricciardelli, and Hitesh Kumar) for working on the products discussed above. Our products would not be possible without a robust and scalable infrastructure. We thank LinkedIn’s Feed Infrastructure partners (Parin Shah, Hassan Khan, and Ali Mohamed) for making this infrastructure easily available. The Feed AI Foundations team at LinkedIn develops tools and frameworks to automate and streamline different aspects of the AI systems that power our products. We thank the Feed AI Foundations team (Zheng Li, Boyi Chen, Ian Ackerman, and Marco Varela Alvarado) for developing the tools and frameworks that make us more productive. We thank Ann Yan and Fangshi Li from the Ranking Infrastructure and Hadoop Development teams for helping us scale our ranking infrastructure. We thank Linda Leung, Fawn Qiu, and Emily Carrolo from our Product Management team for making sure we deliver great product experiences for our members.