Open Source Articles

  • diagram-illustrating-how-data-integration-library-provides-a-small-number-of-connectors-supporting-transfer-protocols-that-cover-the-vast-majority-of-the-use-cases

    Solving the data integration variety problem at scale, with Gobblin

    February 24, 2021

    Co-authors: Chris Li, Kevin Lau, and Subbu Sanka Editor’s Note: Recently, the Apache Software Foundation (ASF) announced Apache® Gobblin™ as a Top-Level Project (TLP). For more information, visit https://gobblin.apache.org/ and https://twitter.com/ApacheGobblin. Introduction Our big data ecosystem is larger than 1 exabyte and growing, while ingesting and...

  • diagram-showing-cycle-of-bias-reinforcement-over-time

    Using the LinkedIn Fairness Toolkit in large-scale AI systems

    February 8, 2021

    Co-authors: Preetam Nandy, Yunsong Meng, Cyrus DiCiccio, Heloise Logan, Amir Sepehri, Divya Venugopalan, Kinjal Basu, and Noureddine El Karoui Introduction LinkedIn’s vision to create economic opportunity for every member of the global workforce would be impossible to realize without leveraging AI at scale. We use AI in our core product offerings to: highlight...

  • school-of-sre-logo-showing-a-gear-wearing-a-graduation-cap

    Open source update: School of SRE

    February 3, 2021

    Co-authors: Akbar KM and Kalyanasundaram Somasundaram Site up and secure is a fundamental element of how we operate, and site reliability engineers (SREs) play a critical role in fulfilling that responsibility. Talent has always been the number one operating priority, and over the last few years, we’ve been running multiple programs to identify, hire, and...

  • gif-showing-smarg-arg-coding-in-action

    Smart Argument Suite: Seamlessly connecting Python jobs

    January 25, 2021

    Co-authors: Jun Jia and Alice Wu Introduction It’s a very common scenario that an AI solution involves composing different jobs, such...

  • FastIngest: Low-latency Gobblin with Apache Iceberg and...

    January 6, 2021

    Co-authors: Zihan Li, Sudarshan Vasudevan, Lei Sun, and Shirshanka Das Data analytics and AI power many business-critical use cases at...

  • coral-a-sql-translation-analysis-and-rewrite-engine

    Coral: A SQL translation, analysis, and rewrite engine for ...

    December 10, 2020

    Co-authors: Walaa Eldin Moustafa, Wenye Zhang, Sushant Raikar, Raymond Lam, Ron Hu, Shardul Mahadik, Laura Chen, Khai Tran, Chris Chen...