Search results for "hadoop"

  • Open-Sourcing the LinkedIn Gradle Plugin and DSL for Apache Hadoop

    August 13, 2015

    I'm proud to announce that the Hadoop Dev Team at LinkedIn has open-sourced the LinkedIn Gradle Plugin for Apache Hadoop ("Hadoop Plugin"), which includes the LinkedIn Gradle DSL for Apache Hadoop ("Hadoop DSL"). You can get the Hadoop Plugin on Github today! A couple of years ago, LinkedIn adopted Gradle as our primary build system. With Gradle, developers can...

  • an-illustration-of-the-distributed-tier-merge

    Distributed tier merge: How LinkedIn tackles stragglers in search index build

    September 27, 2021

    Co-authors: Andy Li and Hongbin Wu Indexing plays the key role in modern search engines for fast and accurate information retrieval, and the ability to swiftly build indexes is crucial for LinkedIn to provide up to date information, such as candidates to recruiters, job posts to members, etc. In some instances, such as if a member profile is missing and...

  • Open Sourcing Dr. Elephant

    April 8, 2016

    We are proud to announce today that we are open sourcing Dr. Elephant, a powerful tool that helps users of Hadoop and Spark understand, analyze, and improve the performance of their flows. We first presented Dr. Elephant to the community last year during the eighth annual Hadoop Summit, a leading conference for the Apache Hadoop community. Hadoop is a framework...

  • hadoopmeetup1

    The Present and Future of Apache Hadoop: A Community Meetup...

    February 21, 2019

    On January 30, Hadoop developers gathered at LinkedIn’s offices in Mountain View to share their latest work, with presentations by...

  • Open Sourcing Photon ML

    June 7, 2016

    Machine learning is a key component of LinkedIn’s relevance-driven products. We use machine learning to train the ranking algorithms...

  • Kafka Ecosystem at LinkedIn

    April 19, 2016

    Apache Kafka is a highly scalable messaging system that plays a critical role as LinkedIn’s central data pipeline. Kafka was developed...