scale Articles

  • new-dualip-project-logo

    DuaLip: Solving extreme-scale linear programs for web applications

    August 6, 2021

    Co-authors: Kinjal Basu, Yao Pan, Rohan Ramanath, Konstantin Salomatin, Amol Ghoting, and S. Sathiya Keerthi Building thriving internet or web products requires optimizing for multiple business metrics. In many applications, these business metrics can move in conflicting directions, and delicately balancing these tradeoffs can be vital for the success of a...

  • gif-showing-new-recruiter-and-jobs-experience

    New Recruiter & Jobs: The largest enterprise data migration at LinkedIn

    July 23, 2021

    Co-authors: Xiaoyang Gu, Xie Lu, and Xiaoguang Wang Introduction In August 2019, we introduced our members and customers to the idea of moving LinkedIn’s two core talent products—Jobs and Recruiter—onto a single platform to help talent professionals be even more productive. This single platform is called the New Recruiter & Jobs. Figure 1: New Recruiter & Jobs...

  • current-architecture-of-the-daily-executive-dashboard-pipeline

    From daily dashboards to enterprise grade data pipelines

    July 20, 2021

    Within a matter of hours of each day beginning, we ingest tens of billions of records from online sources to HDFS, aggregated across 50+ offline data flows, and visualize 40+ metrics across our business lines to support a dashboard providing company leaders with timely and critical business insights. Dubbed the Daily Executive Dashboard (DED), this dashboard...

  • chart-showing-exponential-growth-of-data-metadata-and-compute-on-linkedins-largest-hadoop-cluster

    The exabyte club: LinkedIn’s journey of scaling the Hadoop ...

    May 27, 2021

    Co-authors: Konstantin V. Shvachko, Chen Liang, and Simbarashe Dzinamarira LinkedIn runs its big data analytics on Hadoop. During the...

  • venn-diagram-showing-overlap-of-three-characteristics

    Solving for the cardinality of set intersection at scale...

    April 16, 2021

    Co-authors: Vincent Wang, Siddharth Teotia, Manoj Thakur, and Mayank Shrivastava As our LinkedIn Marketing Solutions Blog recently...

  • diagram-for-variant-assignment-for-a-sample-population

    A/B testing at LinkedIn: Assigning variants at scale

    December 16, 2020

    Co-authors: Alexander Ivaniuk and Weitao Duan Editor’s note: This blog post is the second in a series providing an overview and...