scale Articles

  • chart-showing-exponential-growth-of-data-metadata-and-compute-on-linkedins-largest-hadoop-cluster

    The exabyte club: LinkedIn’s journey of scaling the Hadoop Distributed File System

    May 27, 2021

    Co-authors: Konstantin V. Shvachko, Chen Liang, and Simbarashe Dzinamarira LinkedIn runs its big data analytics on Hadoop. During the last five years, the analytics infrastructure has experienced tremendous growth, almost doubling every year in data size, compute workloads, and in all other dimensions. It recently reached two important milestones. LinkedIn now...

  • venn-diagram-showing-overlap-of-three-characteristics

    Solving for the cardinality of set intersection at scale with Pinot and Theta Sketches

    April 16, 2021

    Co-authors: Vincent Wang, Siddharth Teotia, Manoj Thakur, and Mayank Shrivastava As our LinkedIn Marketing Solutions Blog recently noted, companies and marketers “are once again peering ahead, setting their plans for success in a reshaped business environment.” One of the items businesses rely on to do this are insights, including the estimated reach of an...

  • diagram-for-variant-assignment-for-a-sample-population

    A/B testing at LinkedIn: Assigning variants at scale

    December 16, 2020

    Co-authors: Alexander Ivaniuk and Weitao Duan Editor’s note: This blog post is the second in a series providing an overview and history of LinkedIn’s experimentation platform. The previous post on the history of LinkedIn’s experimentation infrastructure can be found here. Introducing variant assignment Previously on the blog, we’ve shared a look into how...

  • coral-a-sql-translation-analysis-and-rewrite-engine

    Coral: A SQL translation, analysis, and rewrite engine for ...

    December 10, 2020

    Co-authors: Walaa Eldin Moustafa, Wenye Zhang, Sushant Raikar, Raymond Lam, Ron Hu, Shardul Mahadik, Laura Chen, Khai Tran, Chris Chen...

  • test-management-deployment-and-evaluation-workflow-at-linkedin

    Our evolution towards T-REX: The prehistory of...

    September 24, 2020

    Editor’s note: This blog post is the first in a series providing an overview and history of LinkedIn’s experimentation platform. At...

  • title-card

    LIquid: The soul of a new graph database, Part 2

    September 16, 2020

    Co-authors: Scott Meyer, Andrew Carter, and Andrew Rodriguez Editor’s note: This is the second part of a two-part blog series. Part 1...