Kafka Articles

  • change-data-capture

    Open Sourcing Brooklin: Near Real-Time Data Streaming at Scale

    July 16, 2019

    Brooklin - a stateless and distributed service for streaming data in near real-time and at scale - has been running in production at LinkedIn since 2016, powering thousands of data streams and over 2 trillion messages per day. Today, we are pleased to announce the open-sourcing of Brooklin and that the source code is available in our Github repo!  Why Brooklin?...

  • ccfe1

    Introducing Kafka Cruise Control Frontend

    February 7, 2019

    At LinkedIn, Kafka is the de-facto messaging platform that powers diverse sets of geographically-distributed applications at scale. Examples include our distributed NoSQL store (Espresso), stream processing framework (Samza), monitoring infrastructure (InGraphs), and derived data serving platform (Venice). Given these use cases, it’s not surprising that Kafka...

  • samzalogo

    Samza 1.0: Stream Processing at Massive Scale

    November 27, 2018

    We are pleased to announce today the release of Samza 1.0, a significant milestone in the history of the project. Apache Samza is a distributed stream processing framework that we developed at LinkedIn in 2013. Samza became a top-level Apache project in 2014. Fast-forward to 2018, and we currently have over 3,000 applications in production leveraging Samza at...

  • concourse-1

    Concourse: Generating Personalized Content Notifications in...

    May 25, 2018

    Co-authors: Vivek Nelamangala and PJ Xiao   Introduction to Notifications Social media are computer-mediated platforms that facilitate...

  • revisiting-burrow-1

    Revisiting Burrow: Burrow 1.1

    May 15, 2018

    Two and a half years ago, the Data Infrastructure SRE team at LinkedIn introduced Burrow, an advanced way to monitor Apache Kafka...

  • samza-aeon-1

    Samza Aeon: Latency Insights for Asynchronous One-Way Flows

    April 19, 2018

    Co-authors: Max Wolffe and Akhilesh Gupta Introduction You can’t fix something if you don’t know there’s a problem. Measuring and...