Co-authors: Walaa Eldin Moustafa, Wenye Zhang, Adwait Tumbde, Ratandeep Ratti Introduction Over the years, the popularity of Apache Spark at LinkedIn has grown, and users today continue to leverage its unique features for business-critical tasks. Apache Spark allows users to consume datasets using powerful, yet easy-to-use APIs such as the Dataset interface. The...
Spark Articles
-
-
Co-authors: Xuhong Zhang, Chenya Zhang, and Yiming Ma Today, we are announcing a new open source project called Avro2TF. This project provides a scalable Spark-based mechanism to efficiently convert data into a format that can be readily consumed by TensorFlow. With this technology, developers can improve productivity by focusing on building models rather than...
- Topics:
- Spark,
- machine learning,
- TensorFlow,
- Open Source
-
Next Tuesday marks the start of the Spark Summit Conference in San Francisco. This year, LinkedIn engineers and data scientists are presenting four separate talks at the conference, and we’ll be hosting a meetup at our San Francisco office on the final day. All of this is an indication of the significant impact that Apache Spark has had on the way people process...
- Topics:
- Spark,
- Hadoop,
- research,
- Dr. Elephant,
- events
-
This post has been updated to note the release of Pepperdata's Application Profiler, a commercial project based on Dr. Elephant. Last...
- Topics:
- Spark,
- Hadoop,
- Dr. Elephant,
- Open Source
-
Machine learning is a key component of LinkedIn’s relevance-driven products. We use machine learning to train the ranking algorithms...
- Topics:
- Spark,
- Relevance Science,
- Dr. Elephant,
- Open Source,
- relevance,
- Photon ML,
- machine learning
-
We are proud to announce today that we are open sourcing Dr. Elephant, a powerful tool that helps users of Hadoop and Spark understand...
- Topics:
- Spark,
- Hadoop,
- Dr. Elephant,
- Open Source