Augmenting our content moderation efforts through machine learning and dynamic content prioritization
Co-Authors: Abhishek Chandak and Ritish Verma
We recognize that our 1 billion members and their over 10 billion years’ worth of collective knowledge and insights bring tremendous value to the LinkedIn community. That's why we're committed to enabling our members and customers to safely engage and connect with content. Our Trust & Safety team works diligently to keep harmful content off the platform, allowing members to tap into the real-life, meaningful insights and information shared by their peers.
We do this through a multi-layer defense system that includes AI models, member reports, and human reviewers to swiftly identify and remove content that violates our Professional Community Policies. To help our team review content faster and with more focus, we created a new framework for content review prioritization, which uses AI models to score each piece of content in the review queue based on how likely it is to have broken our policies. This helps our reviewers prioritize which content needs immediate attention.
In this blog, we will discuss how this new framework makes it easier for reviewers to find and handle content that needs a human touch, while reducing the average time it takes to detect content that doesn’t align with our policies by 60%. Through a combination of human and technology review systems, this new framework helps ensure that content that doesn’t align to our policies does not make it onto the platform and into our members’ feeds.
How We Designed the New System
Hundreds of thousands of items are ingested in our content review queue every week for human review. One option for prioritizing these reviews is in a First In First Out (FIFO) manner (see Figure 1).
However, this approach has two notable drawbacks. First, not all content that is reviewed by humans violates our policies - a sizable portion is evaluated as non-violative (i.e., cleared). This takes valuable reviewer bandwidth away from reviewing content that is actually violative. Second, when items are reviewed on a FIFO basis, violative content can take longer to detect if it is ingested after non-violative content.
Intelligent review queue prioritization
We developed a new framework for content review prioritization, which uses AI and automation to address these limitations. With this framework, content entering review queues is scored by a set of AI models to calculate the probability that it likely violates our policies. Content with a high probability of being non-violative is deprioritized, saving human reviewer bandwidth and content with a higher probability of being policy-violating is prioritized over others so it can be detected and removed quicker.
The ML models
The new content review prioritization framework uses a set of XGBoost models, to predict the probability of a piece of content being violative or clear. XGBoost provides better recall at set precision for us compared to architectures like TF2-based neural networks as our training data is tabular and there is large variation in violation patterns over time. These models are trained on a representative sample of past human labeled data from the content review queue and tested on another out-of-time sample. We leverage random grid search for hyperparameter selection and the final model is chosen based on the highest recall at extremely high precision (R@P). We use this success metric because LinkedIn has a very high bar for trust enforcements quality so it is important to maintain very high precision.
The models leverage a set of real-time signals, orthogonal to those used by existing AI services for automated content restriction. These models are built on ProML which is the company-wide machine learning productivity platform for training, developing, and serving ML models.
Model Hosting and Decision Layer
The ML models are packed and made available as ML artifacts for consumption. We created a dedicated scoring layer (decision workflow) based on top of ProML which hosts the models and is triggered every time an item enters the review queue. This scorer continuously updates the score of an item in the review queue every time there is a new member report until the item has completed review. The decision workflow is primarily responsible for the following things:
Fetch the features required by the models using Frame from different downstreams and prepare it in a model interpretable format
Run inference on the models, prepare results and return back to upstream i.e., content review queue for consumption
Generate tracking data for analysis, debugging, and precision measurement
This decision layer for serving the ML models results is integrated within the existing review assignment and prioritization tooling at LinkedIn. The content review queue uses an intelligent review assignment framework which uses these scores and few other static parameters to intelligently prioritize human review items.
Advantages of the new approach
Our legacy framework used high-level static buckets with varied SLAs to prioritize the content in the review queue. However, with this new content review prioritization framework, we are using a new and technically complex approach to review prioritization which has the following hallmarks.
This approach to prioritization is completely dynamic where content in the review queue can be moved up or down based on the probability of it being policy-violating.
The scores for a piece of content are updated continuously and the final decision between prioritization or de-prioritization is based on the sum total of all of this information instead of just the information available at a single point in time.
The probability scores are calculated using AI models, which makes the entire framework probabilistic and provides more flexibility for optimization, as compared to a static framework with rigid rules.
The models are triggered at the time a piece of content is reported for review instead of at the time of creation like existing content classification services. As a result, they can make use of additional information that was not available at the time of content creation.
This framework also augments the capacity of human reviewers so it can scale non-linearly with the help of automation and machine intelligence, instead of just expanding linearly as we hire more team members. This will enable us to better scale and keep up with the ever growing content and review volumes on LinkedIn.
Impact
This new content review prioritization framework is able to make auto-decisions on ~10% of all queued content at our established (extremely high) precision standard, which is better than the performance of a typical human reviewer. Due to these savings, we are able to reduce the burden on human reviewers, allowing them to focus on content that requires their review due to severity and ambiguity. With the dynamic prioritization of content in the review queue, this framework is also able to reduce the average time taken to catch policy-violating content by ~60%. Due to this faster identification and subsequent removal of such content, we are able to reduce the number of unique members exposed to violative content significantly each week.
Looking ahead
So far, we have rolled out this new content review prioritization framework for feed posts and comments, and we are working to extend its use into other LinkedIn products. We are also leveraging the foundational capabilities of this framework to enhance other review processes, like reviewer assignment. These steps will help us further strengthen our content moderation practices and ensure LinkedIn’s position as among the most trusted professional networks.
Acknowledgements
Multiple teams across LinkedIn have come together to make this framework a reality and make the platform a safer place for our members. Teams including Trust Data Science, Trust ML Infrastructure, Trust Tools Engineering, Transparency Engineering, Trust Product, Trust AI, Trust and Safety, have all contributed to the development of this framework.