Kafka Articles

  • Nuage1

    Reducing the MTTD and MTTR of LinkedIn’s Private Cloud

    October 14, 2016

    Nuage (French for “cloud”) is what we call LinkedIn’s internal cloud management portal. It allows LinkedIn developers to quickly create new datastores like Kafka topics, Voldemort stores, and Espresso databases, to name just a few in the LinkedIn data centers. The product consists of an HTTP frontend available to LinkedIn developers and a rest.li backend service...

  • streamprocess1

    Stream Processing Hard Problems Part II: Data Access

    August 22, 2016

    This post is the second in a series of posts that discuss some of the hard problems in stream processing. In the previous post, we explored the use of lambda architecture in stream processing and discussed techniques to avoid it. In this post, we’ll focus on one of the main bottlenecks in high scale stream processing applications: “accessing data.” Background...

  • Stream Processing Hard Problems – Part 1: Killing Lambda

    June 27, 2016

    We live in an age where we want to know relevant things happening around the world as soon as they happen; an age where digital content is updated instantly based on our likes and dislikes; an age where credit card fraud, security breaches, device malfunctions and site outages need to be detected and remedied as soon as they happen. It is an age where events are...

  • Kafka Ecosystem at LinkedIn

    April 19, 2016

    Apache Kafka is a highly scalable messaging system that plays a critical role as LinkedIn’s central data pipeline. Kafka was developed...

  • Gobblin Gobbles Camus, Looks Towards the Future

    April 13, 2016

    We shared Gobblin with the open source community a year ago. Since then, we’ve seen increasing interest and adoption among engineers,...

  • Bridging Batch and Streaming Data Ingestion with Gobblin

    September 28, 2015

    Genesis Less than a year ago, we introduced Gobblin, a unified ingestion framework, to the world of Big Data. Since then, we’ve shared...