Kafka Articles

  • cruisecontrol2

    Open Sourcing Kafka Cruise Control

    August 28, 2017

    Apache Kafka's popularity has grown tremendously over the past few years. In fact, LinkedIn's deployment recently surpassed 2 trillion messages per day, with over 1,800 Kafka servers (i.e., brokers). While Kafka has proven to be very stable, there are still operational challenges when running Kafka at such a scale. Brokers fail on a daily basis, which results in...

  • Inception1

    Inception: How LinkedIn Deals with Exception Logs

    December 16, 2016

    Coauthors: Toon Sripatanaskul and Zhengyu Cai In early 2012, the LinkedIn Performance team was trying to build a tool to validate the health of a service after code changes (a project that led us to build EKG, our canary-monitoring system). I was assigned to look into ways to use logs to analyze a service’s health. Back then, we had a script that copied log...

  • ShareAnalytics1

    New Analytics for Sharing on LinkedIn: See Who’s Viewed Your Post

    November 1, 2016

    Coauthor: Bharath Kumarasubramanian   If you’re sharing content on LinkedIn, you’re positioning yourself as a thought leader among the largest group of professionals assembled on the web. With more than 460 million members, LinkedIn is the place you want to be to share your expertise and discuss issues that matter to you with other like-minded professionals. But...

  • Nuage1

    Reducing the MTTD and MTTR of LinkedIn’s Private Cloud

    October 14, 2016

    Nuage (French for “cloud”) is what we call LinkedIn’s internal cloud management portal. It allows LinkedIn developers to quickly...

  • streamprocess1

    Stream Processing Hard Problems Part II: Data Access

    August 22, 2016

    This post is the second in a series of posts that discuss some of the hard problems in stream processing. In the previous post, we...

  • Stream Processing Hard Problems – Part 1: Killing Lambda

    June 27, 2016

    We live in an age where we want to know relevant things happening around the world as soon as they happen; an age where digital...