infrastructure Articles

  • The-new-collaboration-pattern-with-rest-li-and-pegasus

    Empowering our developers with the Nuage SDK

    November 19, 2019

    LinkedIn delivers value to more than 660 million members via thousands of microservices, most of which depend on data infrastructure or mid-tier infrastructure platforms. This means that in order to launch a new application, developers traditionally had to request and set up an online database as a business data source of truth or stream data via Kafka topics,...

  • lag-alert-graphs

    An inside look at LinkedIn’s data pipeline monitoring system

    October 30, 2019

    Co-authors: Krishnan Raman and Joey Salacup   Monitoring big data pipelines often equates to waiting for a long-running batch job to complete and observing the status of the execution. The status can result in “Failed” or “Successful” or even “Incomplete.” From there, it’s the team’s job to understand the impact and troubleshoot the situation to identify a...

  • nuage-architecture

    Solving manageability challenges at scale with Nuage

    September 16, 2019

    Introduction LinkedIn is committed to providing economic opportunities for every member of the global workforce, and we’re growing at a rapid pace. Our platform is built on a collection of large-scale multi-cluster services functioning in harmony to offer a unified product experience to members.  The aim of several backend engineering teams is to allow other...

  • data-hub-logo

    Data Hub: A Generalized Metadata Search & Discovery Tool

    August 14, 2019

    Co-authors: Mars Lan, Seyi Adebajo, Shirshanka Das As the operator of the world’s largest professional network and the Economic Graph,...

  • LinkedIn_Tech_Update

    Building the next version of our infrastructure

    July 23, 2019

    The pursuit of our mission to connect the world’s professionals to make them more productive and successful is deeply dependent on the...

  • change-data-capture

    Open Sourcing Brooklin: Near Real-Time Data Streaming at...

    July 16, 2019

    Brooklin - a distributed service for streaming data in near real-time and at scale - has been running in production at LinkedIn since...