scale Articles

  • LinkedIn-Kafka-ecosystem

    How LinkedIn customizes Apache Kafka for 7 trillion messages per day

    October 8, 2019

    Co-authors: Jon Lee and Wesley Wu Apache Kafka is a core part of our infrastructure at LinkedIn. It was originally developed in-house as a stream processing platform and was subsequently open sourced, with a large external adoption rate today. While many other companies and projects leverage Kafka, few—if any—do so at LinkedIn’s scale. Kafka is used extensively...

  • nuage-architecture

    Solving manageability challenges at scale with Nuage

    September 16, 2019

    Introduction LinkedIn is committed to providing economic opportunities for every member of the global workforce, and we’re growing at a rapid pace. Our platform is built on a collection of large-scale multi-cluster services functioning in harmony to offer a unified product experience to members.  The aim of several backend engineering teams is to allow other...

  • pipeline-cache

    Who Depends On Me? Serving Dependency Queries at Scale

    August 8, 2019

    Co-authors: Yu Li, Szymon Gizecki, and Chinmaya Dattathri To ensure we have significant flexibility in how our teams collaborate, our trunk-based engineering development workflow manages dependencies on a binary level, instead of source level. This requires very efficient and sophisticated management of the resulting dependency graph, and we discussed our...

  • LinkedIn_Tech_Update

    Building the next version of our infrastructure

    July 23, 2019

    The pursuit of our mission to connect the world’s professionals to make them more productive and successful is deeply dependent on the...

  • change-data-capture

    Open Sourcing Brooklin: Near Real-Time Data Streaming at...

    July 16, 2019

    Brooklin - a distributed service for streaming data in near real-time and at scale - has been running in production at LinkedIn since...

  • PartitionConsumer-objects-distribution

    Auto-Tuning Pinot Real-Time Consumption

    July 11, 2019

    Pinot, a scalable distributed columnar OLAP data store developed at LinkedIn, delivers real-time analytics for site-facing use cases...