Kafka Articles

  • ccfe1

    Introducing Kafka Cruise Control Frontend

    February 7, 2019

    At LinkedIn, Kafka is the de-facto messaging platform that powers diverse sets of geographically-distributed applications at scale. Examples include our distributed NoSQL store (Espresso), stream processing framework (Samza), monitoring infrastructure (InGraphs), and derived data serving platform (Venice). Given these use cases, it’s not surprising that Kafka...

  • samzalogo

    Samza 1.0: Stream Processing at Massive Scale

    November 27, 2018

    We are pleased to announce today the release of Samza 1.0, a significant milestone in the history of the project. Apache Samza is a distributed stream processing framework that we developed at LinkedIn in 2013. Samza became a top-level Apache project in 2014. Fast-forward to 2018, and we currently have over 3,000 applications in production leveraging Samza at...

  • concourse-1

    Concourse: Generating Personalized Content Notifications in Near-Real-Time

    May 25, 2018

    Co-authors: Vivek Nelamangala and PJ Xiao   Introduction to Notifications Social media are computer-mediated platforms that facilitate creation and sharing of information, ideas, career interests, and other forms of expression via virtual communities and networks. In the last few years, notifications have been proven to be the mainstay for surfacing social media...

  • revisiting-burrow-1

    Revisiting Burrow: Burrow 1.1

    May 15, 2018

    Two and a half years ago, the Data Infrastructure SRE team at LinkedIn introduced Burrow, an advanced way to monitor Apache Kafka...

  • samza-aeon-1

    Samza Aeon: Latency Insights for Asynchronous One-Way Flows

    April 19, 2018

    Co-authors: Max Wolffe and Akhilesh Gupta Introduction You can’t fix something if you don’t know there’s a problem. Measuring and...

  • venice1

    Venice Hybrid: Doing Lambda Better

    December 20, 2017

    Over the last two years at LinkedIn, I’ve been working on a distributed key-value database called “Venice.” Venice is designed to be a...