Kafka Articles

  • samzalogo

    Samza 1.0: Stream Processing at Massive Scale

    November 27, 2018

    We are pleased to announce today the release of Samza 1.0, a significant milestone in the history of the project. Apache Samza is a distributed stream processing framework that we developed at LinkedIn in 2013. Samza became a top-level Apache project in 2014. Fast-forward to 2018, and we currently have over 3,000 applications in production leveraging Samza at...

  • concourse-1

    Concourse: Generating Personalized Content Notifications in Near-Real-Time

    May 25, 2018

    Co-authors: Vivek Nelamangala and PJ Xiao   Introduction to Notifications Social media are computer-mediated platforms that facilitate creation and sharing of information, ideas, career interests, and other forms of expression via virtual communities and networks. In the last few years, notifications have been proven to be the mainstay for surfacing social media...

  • revisiting-burrow-1

    Revisiting Burrow: Burrow 1.1

    May 15, 2018

    Two and a half years ago, the Data Infrastructure SRE team at LinkedIn introduced Burrow, an advanced way to monitor Apache Kafka consumer clients, in a previous blog post. At the time, we knew it was a gap that needed to be filled, but we were still surprised by how quickly Burrow was adopted by many other companies. Over time, a number of issues were...

  • samza-aeon-1

    Samza Aeon: Latency Insights for Asynchronous One-Way Flows

    April 19, 2018

    Co-authors: Max Wolffe and Akhilesh Gupta Introduction You can’t fix something if you don’t know there’s a problem. Measuring and...

  • venice1

    Venice Hybrid: Doing Lambda Better

    December 20, 2017

    Over the last two years at LinkedIn, I’ve been working on a distributed key-value database called “Venice.” Venice is designed to be a...

  • cruisecontrol2

    Open Sourcing Kafka Cruise Control

    August 28, 2017

    Apache Kafka's popularity has grown tremendously over the past few years. In fact, LinkedIn's deployment recently surpassed 2 trillion...