scale Articles

  • building-a-platform-for-assessments

    Building a platform: Learnings from our pursuit for leverage

    April 3, 2020

    Co-authors: Jefferson Lai, Vinyas Maddi, and Vijay Ramamurthy One of our core engineering principles here at LinkedIn is to create leverage. In practice, what this means is to build software that is easy to reuse by other teams. An effective way we are able to do this is to favor platforms that generalize long-term functionality rather than build inflexible,...

  • schema-management-workflow

    Advanced schema management for Spark applications at scale

    March 25, 2020

    Co-authors: Walaa Eldin Moustafa, Wenye Zhang, Adwait Tumbde, Ratandeep Ratti Introduction Over the years, the popularity of Apache Spark at LinkedIn has grown, and users today continue to leverage its unique features for business-critical tasks. Apache Spark allows users to consume datasets using powerful, yet easy-to-use APIs such as the Dataset interface. The...

  • data-sentinel-logo

    Data Sentinel: Automating data validation

    March 10, 2020

    Co-authors: Arun Swami, Sriram Vasudevan, Sailesh Mittal, Jiefu Zheng, Joojay Huyn, Audrey Alpizar, Changling Huang, Maneesh Varshney, Adrian Fernandez Data’s value is best realized when prepared and treated correctly. However, when you’re working with data at an extensive scale, it’s not as easy to make sure that every data set has been cleaned and validated....

  • stages-in-continuous-deployment

    Learnings from the journey to continuous deployment

    October 18, 2019

    As an engineer, your goal is for every commit to seamlessly land in production and provide a delightful experience for your customers....

  • LinkedIn-Kafka-ecosystem

    How LinkedIn customizes Apache Kafka for 7 trillion...

    October 8, 2019

    Co-authors: Jon Lee and Wesley Wu Apache Kafka is a core part of our infrastructure at LinkedIn. It was originally developed in-house...

  • nuage-architecture

    Solving manageability challenges at scale with Nuage

    September 16, 2019

    Introduction LinkedIn is committed to providing economic opportunities for every member of the global workforce, and we’re growing at...