Building Trust and Combating Abuse On Our Platform
At LinkedIn, trust is the cornerstone for building meaningful connections and professional relationships. Our members rely on us to create an environment on our platform where they can safely learn and grow in their careers. As part of this responsibility, we are constantly looking for innovative ways to prevent policy-violating behavior such as harassment, spamming, fake accounts, and other activities that undermine the positive and respectful interactions our community strives to uphold. By leveraging cutting-edge technologies, machine learning algorithms, and a dedicated team, we remain committed to ensuring a secure and trustworthy space for professionals to connect, share insights, and foster their career journeys.
In this blog post, we discuss how we are harnessing AI to help us with abuse prevention and share an overview of our infrastructure and the role it plays in identifying and mitigating abusive behavior on our platform. We also outline the complex systems that underpin our anti-abuse efforts, discussing the challenges and solutions we have designed along the way.
Real-time risk assessment and mitigation
One key aspect of LinkedIn's Trust Machine Learning (ML) infrastructure is its ability to perform real-time risk assessment and mitigation. Our commitment to trust and safety goes beyond reactive measures; we strive to prevent abusive behavior before it even happens. Our engineering teams understand the importance of proactive defense, and we’ve harnessed the power of AI to make it a reality. Let’s look into the critical modules that are needed to build this type of system.
Understanding the intent
Our journey begins with understanding the intent of the member. LinkedIn is a diverse platform that connects one billion professionals with varying backgrounds and motivations. To effectively combat abuse, we need to decipher the subtleties of their behavior and differentiate between genuine interactions and potential threats.
Imagine a scenario where an experienced engineer reaches out to a recent graduate offering mentorship. Interactions like this are the core of LinkedIn’s purpose, fostering professional growth. However, when sent repeatedly and indiscriminately to multiple members, the same initial message could be seen as spam or potentially abusive. Our challenge lies in distinguishing between these scenarios instantaneously. To provide a safer and more trusted platform, we must both identify and anticipate abuse to prevent harm to individual members and our LinkedIn community. This task is multifaceted, involving the recognition of various abuse vectors and the detection of potential damage to members and the network as a whole.
Collecting signals
Signals are vital data points that provide insights into member activities. These signals encompass a rich tapestry of actions, from member-reported abuse to content interactions. One of the challenges we encounter in signal collection is recognizing that abusive behavior is not one-dimensional. It can manifest in numerous ways, often hiding behind seemingly innocuous actions. What may appear as a harmless interaction in isolation can take on a different and potentially harmful context when viewed in conjunction with other signals. Hence, gathering a diverse set of signals allows us to form a holistic view of the intent.
Collecting signals is not just about quantity; it's about ensuring the diversity and quality of the data, and ensuring that the data is relevant, accurate, and reflective of the real-world activities of the member. We need systems to ingest, process, and analyze the data efficiently. This entails rigorous data curation to filter out noise and focus on meaningful signals and processes and enables us to make timely and accurate decisions, even in a high-velocity digital environment.
Adversarial attacks
Bad actors seek to exploit vulnerabilities within online platforms for their gain, often going to great lengths to avoid detection. These clandestine tactics are commonly referred to as "adversarial attacks," which are multi-faceted and can manifest in various forms, each designed to manipulate the platform without triggering alarms. Examples of such attacks include:
- Automated spamming: deploying bots to flood the platform with spammy content
- Fake profile creation: creating fake profiles for non-existent people to perform malicious activities like conducting scams, or engaging in online harassment
These attacks erode trust and can cause significant harm to individuals and the community at large.To effectively combat these advanced threats, we must employ a combination of specialized adversarial models and incident management rules. These models and rules are designed to think like an adversary, monitoring for patterns of behavior that deviate significantly from normal member interactions. By adopting an adversarial mindset, we can anticipate the tactics employed by bad actors and proactively detect their activities. We design our systems to be robust and adaptable, continually evolving to stay ahead of emerging threats; along with a dynamic approach to technology and strategy, ensuring that our defenses are resilient in the face of ever-evolving attack vectors.
Decision making
Leveraging the collected signals, we need to build a framework that bridges this multitude of signals, machine learning modules, business rules, and policy enforcement into a dynamic, responsive system.
At the core of inference at scale lies the fusion of ML with a wealth of data. ML models, meticulously trained on diverse and high-quality datasets, are the linchpin of our Abuse detection efforts. Fueled by signals gathered from user behavior, content integrations, and contextual information, these models evaluate user actions with precision and speed. ML models analyze the collected signals to generate scores, indicating the likelihood of an action being abusive or in violation of our LinkedIn platform policies.
While ML models are indispensable in recognizing patterns and deviations, our framework also integrates with the business rules engine. These rules serve as a crucial layer of policy enforcement, ensuring specific guidelines and ethical standards are upheld. These rules are designed to complement the output of ML models. They provide an additional layer of scrutiny, aligning member actions with data-driven insights, community standards, and ethical considerations.
We further leverage proprietary algorithms to weigh and evaluate signals, providing a more nuanced understanding of the intent. These algorithms consider the diversity and context of signals to make informed decisions. We need to ensure that actions in violation of LinkedIn’s guidelines are promptly addressed. This includes taking measures such as issuing warnings, restricting access, or suspending accounts as necessary.
Feedback loop
In the landscape of abuse prevention, perpetual evolution is not a luxury but a necessity. Our systems must remain dynamic, responsive, and equipped with a robust feedback loop to outpace the ever-changing threat landscape. Stagnation is not an option; it implies vulnerability. Establishing a responsive feedback loop is imperative to ensure our anti-abuse systems can adapt to shifting behaviors and emerging threats at scale.
As we entrust complex and intelligent systems with user intent decisions, the importance of an active feedback loop becomes evident. Standing still in the realm of abuse prevention is synonymous with regression. The feedback loop serves as a critical component of a dynamic defense strategy, constantly monitoring and aggregating data from abuse reports, member feedback, and reviewer input. This wealth of information becomes the foundation on which we craft enhancements. By scrutinizing patterns with abuse data, we pinpoint emerging trends, allowing us to fine-tune our models and systems in real-time. This iterative process guarantees our defense mechanisms' resilience, effectiveness, and adaptability.
Integral to this feedback loop are our human reviewers, an invaluable asset in our defensive strategy. While ML models excel at pattern recognition, the nuanced understanding of human intuition remains unparalleled. Reviewers play a pivotal role in dissecting intricate and complex scenarios that may elude algorithmic detection. Their insights, grounded in contextual comprehension, refine our models and enable them to discern genuine interactions from potential threats with greater precision. In this collaborative synergy, human reviewers contribute to our models' continual training and improvement, ensuring they evolve alongside the dynamic landscape they safeguard.
Inference at scale
To handle this colossal volume of data efficiently, we need a system that ensures minimal latency while delivering accurate abuse assessments. This scalability empowers us to maintain the responsiveness required to counter abuse at scale, thereby creating a trusted space for professionals to connect, learn and grow.
Introducing CASAL
In the dynamic landscape of LinkedIn, innovation and adaptability are desirable traits and essential lifelines. To uphold this commitment, our anti-abuse systems have had to evolve in lockstep with the platform’s growth, the emergence of new products, and the ever-ingenious tactics employed by malicious actors.
Before CASAL, our anti-abuse scoring system was monolithic in nature. The monolith served as a sentinel, guarding against a multitude of abuse vectors within a single system. It was a solid effort that carried us through prior stages, but as our platform expanded and our team grew, it became evident that we should adapt. With new products being launched, there was an upsurge in novel adversarial attacks and operational overhead. The monolithic system, while commendable, was no longer equipped to handle the scale and complexity of these challenges effectively.
Recognizing the need for a more agile, scalable, and specialized approach, we embarked on a transformative journey that led to the inception of the CASAL: Centralized Abuse Scoring As a Library framework. It was a paradigm shift in how we empowered our teams in Trust engineering to deploy the defenses against the abuse vectors in consideration. It represents the culmination of our expertise, innovation, and unwavering dedication to trust and safety. (Scoring: In machine learning, scoring is the process of applying an algorithmic model built from a historical dataset to a new dataset to uncover practical insights that will help solve a business problem.)
The cornerstone of our anti-abuse architecture at Linkedin is the CASAL framework. CASAL is the entry point that allows us to support abuse detection in synchronous (online) and asynchronous (nearline) request processing. In essence, this translates to enabling real-time and near-real-time defense decision workflows. The beauty of CASAL lies in its ability to empower the teams to author distinct decision workflows, encapsulating the logical flow for execution in isolation. This approach helps us avoid “noisy neighbor” issues and significantly reduces operational overhead.
Customized decision workflows: Within CASAL, we’ve harnessed the power of trigger points for specific use cases like fake account detection, account takeover detection, content moderation, anti-scraping, and more. These trigger points enable us to invoke the corresponding decision workflow tailored to the unique requirements of each abuse detection scenario.
Feature-rich decision making: For each individual workflow, we rely on predefined sets of features that are essential for making informed decisions. These features are the building blocks for our scoring process and are fetched in parallel from various data sources across the expansive LinkedIn ecosystem (e.g., Espresso, Venice, Rest.li services, Couchbase). Leveraging our underlying microservices architecture, we have designed the parallel feature retrieval to optimize the latency of our scoring requests. It’s crucial for us to detect abuse in line with the request, across different product surfaces, without degrading the member experience.
Diverse signals and features: The signals transformed into useful features span a wide spectrum of data, encompassing entity reputation, counters, member behavioral patterns, requests from known fraudulent entities, and much more. Collecting this extensive array of signals, or features, is a critical step for every request as it forms the foundation for our abuse detection.
Caching for efficiency: To optimize efficiency, we have implemented caching for features that remain constant over the member’s lifecycle or change rarely, such as the LinkedIn account creation date. This helps us avoid unnecessary calls to data sources, reducing latency and operational overhead.
Scoring modes - CURRENT & PROPOSED: Once these features are collected, CASAL initiates the scoring process, bifurcating it into two modes: CURRENT and PROPOSED. CURRENT mode involves scoring the request and taking immediate action based on whether it’s deemed abusive or not. On the other hand, PROPOSED mode is reserved for scoring requests with newer experimental defenses without taking immediate action. This mode empowers our team to deploy and test newer defense algorithms, conducting experiments to gauge their effectiveness. The decision to promote these defenses to the CURRENT mode is based on the confidence of the experimentation results.
ML model inference: At the heart of our anti-abuse systems lies the formidable capability of ML model inference. The pivotal step is where we harness the power of AI to classify the requests effectively. To do this, we employ a meticulously crafted dependency graph, incorporating various model architectures tailored to our needs.
Our collection of ML models spans a diverse spectrum, from the tried-and-true logistic regression, to powerful tree-based models like XGBoost and isolation forest and cutting-edge deep-learning approaches like sequence models and graph neural networks. Each of these models is custom-tailored and trained to address specific abuse vectors that our system must detect and classify. The result is a symphony of AI-driven insights, each model generating a score and potential classification label. These scores, derived from a multitude of models, become the cornerstone of our decision-making process, enabling us to assess if the incoming request is abusive or not (and also if it violates our policies or terms of service).
It’s essential to note that these ML models are not static entities. Instead, they are continuously evolving, much like the ever-changing landscape of abuse tactics. We train these models periodically, aligning their capabilities with the frequency of shifts underlying abuse vectors. This adaptive approach ensures that our AI is always at the cutting edge, ready to tackle emerging threats. CASAL provides the framework to deploy newer version ML model deployments and perform inference at scale continuously, helping us keep our workflows updated.
Business rules: While ML model inference is a powerful tool, we complement it with another critical element: business rules. These rules bridge AI-driven insights and human expertise, allowing us to make nuanced decisions. Think of them as a set of conditions in a “when-then” format, carefully crafted to consider a combination of features and signals collected during the process. We leverage an open source business rules management system called DROOLS to author them.
Some of these rules draw from historical evidence informed by past encounters with adversarial attacks or known malicious entities. These rules evolve over time, with some designed for immediate impact on ongoing attacks, while others serve as steadfast guardians of our platform’s policies. They are tailored for each abuse vector, ensuring precise classification.
Our team is at the helm of authoring these rules, constantly monitoring our defenses and identifying gaps. We perform rigorous data analysis to craft a new rule when a gap is detected. The power of these rules lies in their speed, as they go into effect immediately without the need for time-consuming deployment changesets. These rules are authored/modified and evolve in a version-based manner. They are asynchronously updated in a datastore which does not require stop-the-world deployment and becomes effective within a few minutes.
The CURRENT and PROPOSED modes we discussed earlier come into play here. In the PROPOSED mode, we define new rules and conduct experiments to validate their efficacy, closely monitoring for exceptions and false positives. This iterative process ensures that our defenses are both robust and agile, capable of responding swiftly to emerging threats. Eventually, after a certain confidence level is reached, we promote these rules to CURRENT mode, wherein we start taking action on the requests.
Experimentation: We need a control plane to launch new defense strategies using newer features, derivations of those features, ML models, and business rules in a controlled environment. We leverage the LinkedIn internal experimentation framework called TREX (Targeting, Ramping, and Experimentation platform) to perform this controlled ramp. TREX allows us to enable targeting, ramping, and experimentation, where a team member can quickly set up an experiment against the targeted member group, ramp the new capability, and quantify its impact in a scientific and controlled manner across all LinkedIn products. TREX allows easy design and deployment of experiments, but it also provides automatic analysis, which is crucial in increasing the adoption of A/B experiments.
Enforcement: Once the scoring requests have traversed the ML model inference and business rule phases, it’s time to determine the appropriate action to take. The wealth of information collected during the process, including scores and classification labels, guides us in making these crucial decisions. For requests that are unequivocally identified as abusive and violate our platform policies, we take immediate action. This might involve enforcing challenges (e.g., CAPTCHA) to distinguish between a bot or human, issuing warnings, sending the entity to human review queue to perform further manual analysis, and in severe cases restricting the account. The aim is not only to protect our members but also to maintain the integrity of the LinkedIn community.
Conversely, when a request is classified as non-abusive with high confidence, we allow it to proceed without interference. Striking the balance between proactive defense and member experience is at the core of our approach.
Tracking: Tracking and monitoring the decisions made within our anti-abuse systems are integral to our continuous improvement efforts. To achieve this, we leverage Kafka messages, a robust and scalable event streaming platform. Kafka allows us to efficiently record and transmit decision data in real-time along with the blueprint of the scoring request including the features collected, ML model scores, activated business rules, and the final set of actions, ensuring that we maintain visibility into our system’s performance.
These Kafka messages serve as a comprehensive record of our decisions, enabling us to conduct thorough post-analysis, identify patterns, and assess the effectiveness of our defenses. This data-driven approach informs our strategy for further enhancements, ensuring that our systems evolve to stay ahead of emerging threats. We also use this dataset to integrate with our in-house built MLOps platform to monitor the feature drift detection, model scores, anomaly detection, etc.
Human review: While our AI-driven models and business rules are highly effective, we recognize the value of human judgment, particularly in nuanced and complex situations. Human reviewers play a pivotal role in our framework. They are equipped with the expertise to perform manual reviews, diving deep into cases that demand a human touch. When a request falls into a gray area or when there’s uncertainty in the classification, human reviewers step in. They provide essential insights, adding an extra layer of scrutiny to ensure that we strike the right balance between automation and human intervention. This collaborative approach safeguards against false positives and ensures that we make well-informed decisions that uphold our platform’s trust and safety standards.
Labeling: A crucial aspect of our system is the meticulous process of labeling. This involves categorizing cases as “true positives” when the system correctly identifies abusive behavior or “false positives” when the system flags non-abusive behavior incorrectly. Labeling holds immense significance in our quest to maintain the highest standards by helping assess the accuracy and performance of our ML models. We gain invaluable insights into how well our models and rules are functioning. This data-driven evaluation allows us to fine-tune our defenses for optimal results, continually enhancing our ability to detect and combat abusive behavior.
Furthermore, the labeled data plays a pivotal role in driving model refinement. It provides a rich source of information for training our models to adapt to the ever-evolving landscape of abuse tactics and user behavior patterns.
Model training: Leveraging the labeled data generated from the true positives and false positives, we need to establish a process that is a continuous cycle of learning and adaptation. Our ML models are trained to analyze this data, identifying nuances and patterns that might escape the human eye. By revisiting and refining the models in the offline environment, we ensure that they are equipped to tackle the latest abuse tactics and user behavior trends–enabling us to respond swiftly.
As CASAL is primarily designed for teams to leverage in performing inference in online (REST: sync & async) and nearline (Kafka) environments, we offload the ML model training to be done in offline environments using LinkedIn Azkaban workflow manager & Hadoop clusters. Through offline ML model training, we harness this data to ensure our models remain at the forefront of abuse detection.
Offline ML model training isn't just about keeping up with the change; it is about optimizing the performance. Through rigorous experimentation and fine-tuning, we aim to maximize the precision and recall of our models. We meticulously evaluate different algorithms, features, and hyperparameters to achieve the best possible results.
Conclusion
LinkedIn's anti-abuse framework, CASAL, serves as a vanguard against sophisticated adversaries, reinforcing our commitment to creating a secure and respectful environment for professionals to connect, learn, and grow.
CASAL is the result of a collaborative effort between our engineering and AI teams, data scientists, and domain experts. By fostering a culture of cross-functional collaboration and iterative development, we continuously enhance our algorithms and models to adapt to emerging threats and evolving member needs. This collaborative approach ensures that our Trust ML infrastructure remains effective and robust, further strengthening LinkedIn's reputation as a trusted platform that prioritizes the security and satisfaction of its members.
In this engineering blog post, we have delved into the Trust ML infrastructure at LinkedIn, highlighting its role in reinforcing the company's reputation as a trusted industry leader. By showcasing the advanced machine learning techniques, real-time risk assessment capabilities, scalability, and collaborative approach to continuous improvement, we have demonstrated how LinkedIn's engineering efforts in building and optimizing the Trust ML infrastructure contribute to creating a safe and trustworthy platform for our members. LinkedIn solidifies its position as a trusted destination for professionals worldwide through this commitment to engineering excellence.
Acknowledgments
We would like to extend our heartfelt acknowledgments to the dedicated team who have been the driving force behind our anti-abuse platform success. This is a massive collaborative effort of our core team, comprising brilliant minds from various domains, including Anti-Abuse Infra, Anti-Abuse AI, Data Science (DS), Incident Management, Product Management, Content Review, Productive Machine Learning platform (ProML) and beyond. Their relentless commitment to innovation, problem-solving, and a shared vision of creating a safer online space for professionals worldwide has been truly commendable.
I would also like to extend Xiaofeng Wu, Shane Ma, Hongyi Zhang, James Verbus, Bruce Su, Bing Wang, Abhishekh Bajoria, Katherine Vaiente, Francisco Tobon, Jon Adams, Will Cheng, and Kayla Guglielmo for helping review this blog and provide valuable feedback.