Data Articles

  • pegasus-data-language

    Pegasus Data Language: Evolving schema definitions for data modeling

    November 19, 2020

    Pegasus Data Schema (PDSC) is a Pegasus schema definition language that has been used for data modeling with Rest.li services for years. It's the underlying language that helps define data models, describe the data returned by REST endpoints, and generate derivative schemas for other uses, such as XML schemas and various database schemas. However, writing PDSC...

  • open-sourcing-dagli-for-faster-and-easier-machine-learning

    Dagli: Faster and easier machine learning on the JVM, without the tech debt

    November 10, 2020

    In recent years, we’ve been fortunate to see a growing number of excellent machine learning tools, such as TensorFlow, PyTorch, DeepLearning4J, and CNTK for neural networks, Spark and Kubeflow for very-large-scale pipelines, and scikit-learn, ML.NET, and the recent Tribuo for a wide variety of common models. However, models are typically part of an integrated...

  • architecture-diagram-of-magnet

    Magnet: A scalable and performant shuffle architecture for Apache Spark

    October 21, 2020

    Co-authors: Min Shen, Chandni Singh, Ye Zhou, and Sunitha Beeram At LinkedIn, we rely heavily on offline data analytics for data-driven decision making. Over the years, Apache Spark has become the primary compute engine at LinkedIn to satisfy such data needs. With its unique features, Spark empowers many business-critical tasks at LinkedIn, including data...

  • pensieve-an-embedding-feature-platform

    Pensieve: An embedding feature platform

    October 14, 2020

    Co-authors: Benjamin Le, Daniel Gmach, Aman Grover, Roshan Lal, Jerry Lin, Austin Lu, Qingyun Wan, and Leighton Zhang Feature...

  • sketching-out-what-a-heterogeneous-social-network-looks-like

    Building a heterogeneous social network recommendation...

    October 6, 2020

    Co-authors: Parag Agrawal, Ankan Saha, Yafei Wang, Aastha Nigam, and Eric Lawrence Figure 1: A heterogeneous social network LinkedIn’s...

  • title-card

    Helping members discover communities around interests

    September 17, 2020

    Co-authors: Chiachi Lo, Bohong Zhao, and Elina Lin When we launched a major redesign of LinkedIn’s mobile application and desktop web...