Co-authors: Erik Krogen and Min Shen In March 2015, LinkedIn’s Big Data Platform team experienced a crisis. As the team was preparing to head home for the day, signs of trouble began trickling in: our internal users were reporting that their applications were stalling or timing out. Job queues were backing up, and SLAs would be missed. A bit of investigation...
Hadoop Articles
-
- Topics:
- Testing,
- Hadoop,
- Open Source
-
Co-authors: Carl Steinbach and Vasanth Rajamani Big challenges in the big data ecosystem At LinkedIn, we have a number of challenges managing data in our complex data ecosystem. Changes to our infrastructure are often necessary to make progress, but they are difficult to accomplish without an expensive, large-scale, coordinated effort. Analytics processing...
-
Next Tuesday marks the start of the Spark Summit Conference in San Francisco. This year, LinkedIn engineers and data scientists are presenting four separate talks at the conference, and we’ll be hosting a meetup at our San Francisco office on the final day. All of this is an indication of the significant impact that Apache Spark has had on the way people process...
- Topics:
- Spark,
- Hadoop,
- research,
- Dr. Elephant,
- events
-
This post has been updated to note the release of Pepperdata's Application Profiler, a commercial project based on Dr. Elephant. Last...
- Topics:
- Spark,
- Hadoop,
- Dr. Elephant,
- Open Source
-
About a year ago, we open sourced Gobblin, a universal data ingestion framework that aimed to solve data integration challenges faced...
- Topics:
- Hadoop,
- Big Data,
- Open Source,
- Apache Kafka,
- Data Ingestion,
- Distributed Systems,
- ETL,
- Gobblin
-
We shared Gobblin with the open source community a year ago. Since then, we’ve seen increasing interest and adoption among engineers,...
- Topics:
- Hadoop,
- Data Ingestion,
- Gobblin,
- Kafka,
- Open Source