• Open Sourcing Kafka Monitor

    May 25, 2016

    Apache Kafka has become a standard messaging system for large-scale, streaming data. In companies like LinkedIn it is used as the backbone for various data pipelines and powers a variety of mission-critical services. It has become a core component of a company’s infrastructure that should be extremely robust, fault-tolerant and performant. In the past, Kafka Site Reliability Engineers (SRE) have relied on metrics reported by Kafka servers (e.g. bytes-in rate,...

  • Kafkaesque Days at LinkedIn – Part 1

    May 24, 2016

    This is the first post in a blog series adapted from my talk at the inaugural Kafka Summit. Apache Kafka is the backbone for various data pipelines and asynchronous messaging at LinkedIn and beyond. In fact, we were excited to learn at the Kafka Summit last month that a number of other big adopters like Netflix and Microsoft have hit similar levels of scale in terms of message volume and deployment sizes, firmly establishing Kafka in the Four Comma Club. While Kafka is an...

  • Speed, Performance, and Public Profile

    May 17, 2016

    Co-authors: Gaurav Phapale and Oliver Tse A public profile page is a guest-accessible LinkedIn member’s profile page. When you visit the same profile as logged-in user, it’s called a member profile. For example, when you visit Jeff’s profile, you see member profile when you are logged in to LinkedIn and you see public profile when you are logged out of LinkedIn. While both these pages show profile information of the same LinkedIn member, they have slightly different content and...

  • Media infrastructure with Ambry

    Introducing and Open Sourcing Ambry

    May 16, 2016

    Media content has become ubiquitous around the web and almost all of Linkedin's new features interact with media in some form or the other. Profile photos...

  • Rupesh Gupta

    Getting to Know Rupesh Gupta

    May 12, 2016

    LinkedIn wouldn't be the company it is today without the engineers who built it. We have no shortage of talented individuals in technical roles across the...

  • Automating Large-Scale Application Build

    May 11, 2016

    Around the end of 2012, LinkedIn decided to move away from retail data centers to wholesale ones that are built and maintained by LinkedIn. With...