Kafka Articles

  • venice1

    Venice Hybrid: Doing Lambda Better

    December 20, 2017

    Over the last two years at LinkedIn, I’ve been working on a distributed key-value database called “Venice.” Venice is designed to be a significant improvement to Voldemort Read-Only for serving derived data. In late 2016, Venice started serving production traffic for batch use cases that were very similar to the existing uses of Voldemort Read-Only. In the time...

  • cruisecontrol2

    Open Sourcing Kafka Cruise Control

    August 28, 2017

    Apache Kafka's popularity has grown tremendously over the past few years. In fact, LinkedIn's deployment recently surpassed 2 trillion messages per day, with over 1,800 Kafka servers (i.e., brokers). While Kafka has proven to be very stable, there are still operational challenges when running Kafka at such a scale. Brokers fail on a daily basis, which results in...

  • Inception1

    Inception: How LinkedIn Deals with Exception Logs

    December 16, 2016

    Coauthors: Toon Sripatanaskul and Zhengyu Cai In early 2012, the LinkedIn Performance team was trying to build a tool to validate the health of a service after code changes (a project that led us to build EKG, our canary-monitoring system). I was assigned to look into ways to use logs to analyze a service’s health. Back then, we had a script that copied log...

  • ShareAnalytics1

    New Analytics for Sharing on LinkedIn: See Who’s Viewed...

    November 1, 2016

    Coauthor: Bharath Kumarasubramanian   If you’re sharing content on LinkedIn, you’re positioning yourself as a thought leader among the...

  • Nuage1

    Reducing the MTTD and MTTR of LinkedIn’s Private Cloud

    October 14, 2016

    Nuage (French for “cloud”) is what we call LinkedIn’s internal cloud management portal. It allows LinkedIn developers to quickly...

  • streamprocess1

    Stream Processing Hard Problems Part II: Data Access

    August 22, 2016

    This post is the second in a series of posts that discuss some of the hard problems in stream processing. In the previous post, we...