infrastructure Articles

  • How LIquid Connects Everything So Our Members Can Do Anything

    May 2, 2023

    Imagine a tool that can store and connect all the information you need to make decisions and solve problems. Most people would say it’s nice to think about, but not yet possible. The good news is this tool already exists - and it’s called a graph database. At LinkedIn, technologies like graph databases are essential to powering today's platform, while being...

  • Scaling Salt for Remote Execution to support LinkedIn Infra growth

    April 18, 2023

    At LinkedIn, site engineers like to automate operational tasks at various infrastructure layers to minimize manual interventions, which can scale well and be easy to operate. Certain automations are performed via onDemand job executions. LinkedIn engineers have been using Salt, a Python-based, open source software, for automating tasks at various infrastructure...

  • Reducing Apache Spark Application Dependencies Upload by 99%

    March 9, 2023

    Co-authors: Shu Wang, Biao He, and Minchu Yang At LinkedIn, Apache Spark is our primary compute engine for offline data analytics such as data warehousing, data science, machine learning, A/B testing, and metrics reporting. We execute nearly 100,000 Spark applications daily in our Apache Hadoop YARN (more on how we scaled YARN clusters here). These applications...

  • Unifying Messaging Experiences across LinkedIn

    January 26, 2023

    Co-authors: Michele Ursino and Joe Xue Introduction At LinkedIn, we believe that an opportunity can arise from just one conversation,...

  • Accelerating Code Delivery By 97% With Yarn Workspaces

    December 15, 2022

    As teams and applications experience growth, it’s critical to adopt architectures that optimize for clear code ownership, build...

  • Co-authors: Kenneth Tay and Xiaofeng Wang At Linkedin, we constantly evaluate the value our products and services deliver, so that we...