Hadoop Articles

  • dali-datasets-feature-image

    Dali Views: Functions as a Service for Big Data

    November 9, 2017

    Co-authors: Carl Steinbach and Vasanth Rajamani Big challenges in the big data ecosystem At LinkedIn, we have a number of challenges managing data in our complex data ecosystem. Changes to our infrastructure are often necessary to make progress, but they are difficult to accomplish without an expensive, large-scale, coordinated effort. Analytics processing...

  • sparksummit2

    Spark Summit 2017: Research, Open Source, and Community

    June 2, 2017

    Next Tuesday marks the start of the Spark Summit Conference in San Francisco. This year, LinkedIn engineers and data scientists are presenting four separate talks at the conference, and we’ll be hosting a meetup at our San Francisco office on the final day. All of this is an indication of the significant impact that Apache Spark has had on the way people process...

  • drelephant2

    A Checkup with Dr. Elephant: One Year Later

    March 6, 2017

    This post has been updated to note the release of Pepperdata's Application Profiler, a commercial project based on Dr. Elephant. Last April, we announced the first open source release of Dr. Elephant, a performance monitoring and tuning service for Hadoop and Spark jobs. That announcement marked the culmination of two years of internal development work and more...

  • Announcing Gobblin 0.7.0: Going Beyond Ingestion

    June 29, 2016

    About a year ago, we open sourced Gobblin, a universal data ingestion framework that aimed to solve data integration challenges faced...

  • Gobblin Gobbles Camus, Looks Towards the Future

    April 13, 2016

    We shared Gobblin with the open source community a year ago. Since then, we’ve seen increasing interest and adoption among engineers,...

  • Open Sourcing Dr. Elephant

    April 8, 2016

    We are proud to announce today that we are open sourcing Dr. Elephant, a powerful tool that helps users of Hadoop and Spark understand...