Kafka Articles

  • lag-alert-graphs

    An inside look at LinkedIn’s data pipeline monitoring system

    October 30, 2019

    Co-authors: Krishnan Raman and Joey Salacup   Monitoring big data pipelines often equates to waiting for a long-running batch job to complete and observing the status of the execution. The status can result in “Failed” or “Successful” or even “Incomplete.” From there, it’s the team’s job to understand the impact and troubleshoot the situation to identify a...

  • LinkedIn-Kafka-ecosystem

    How LinkedIn customizes Apache Kafka for 7 trillion messages per day

    October 8, 2019

    Co-authors: Jon Lee and Wesley Wu Apache Kafka is a core part of our infrastructure at LinkedIn. It was originally developed in-house as a stream processing platform and was subsequently open sourced, with a large external adoption rate today. While many other companies and projects leverage Kafka, few—if any—do so at LinkedIn’s scale. Kafka is used extensively...

  • FollowFeed-flow

    Auditing Content Features in FollowFeed

    August 27, 2019

    The LinkedIn feed relies on a ranked list of the most relevant content for a member. More than 80% of the feed is organic content created by people, companies, or groups that a member is following; the rest consists of recommendations such as jobs, articles, or ads. All organic content in the Linkedin feed is powered by FollowFeed. FollowFeed has two main...

  • change-data-capture

    Open Sourcing Brooklin: Near Real-Time Data Streaming at...

    July 16, 2019

    Brooklin - a distributed service for streaming data in near real-time and at scale - has been running in production at LinkedIn since...

  • ccfe1

    Introducing Kafka Cruise Control Frontend

    February 7, 2019

    At LinkedIn, Kafka is the de-facto messaging platform that powers diverse sets of geographically-distributed applications at scale....

  • samzalogo

    Samza 1.0: Stream Processing at Massive Scale

    November 27, 2018

    We are pleased to announce today the release of Samza 1.0, a significant milestone in the history of the project. Apache Samza is a...