Co-authors: Zihan Li, Sudarshan Vasudevan, Lei Sun, and Shirshanka Das Data analytics and AI power many business-critical use cases at LinkedIn. We need to ingest data in a timely and reliable way from a variety of sources, including Kafka, Oracle, and Espresso, bringing it into our Hadoop data lake for subsequent processing by AI and data science pipelines. We...
Hadoop Articles
-
- Topics:
- Stream Processing,
- Hadoop,
- Data,
- batch processing,
- Open Source,
- Gobblin,
- Kafka
-
Co-authors: Sandhya Ramu and Vasanth Rajamani For companies and organizations, failure tends to be far more illuminating than success and the lingering effects of a failure can be harmful if the team moves too quickly and does not resolve the issue in a thorough and transparent manner. We recently ran into a large incident that involved data loss in our big data...
- Topics:
- Performance,
- Hadoop,
- data center,
- Data
-
Co-authors: Cong Gu, Abin Shahab, Chen Qiang, and Keqiu Hu Editor's note: This blog has been updated. LinkedIn AI has been traditionally Hadoop/YARN based, and we operate one of the world’s largest Hadoop data lakes, with over 4,500 users and 500PB of data. In the last few years, Kubernetes has also become very popular at LinkedIn for Artificial Intelligence (AI...
- Topics:
- Hadoop,
- Security,
- Open Source
-
Co-authors: Christian Mathiesen and Jie Zhang Your LinkedIn profile is intended to be a representative picture of your professional...
- Topics:
- Hadoop,
- Product Design,
- ESPRESSO
-
On January 30, Hadoop developers gathered at LinkedIn’s offices in Mountain View to share their latest work, with presentations by...
- Topics:
- Hadoop,
- Data,
- Open Source,
- events
-
Co-authors: Jonathan Hung, Keqiu Hu, and Anthony Hsu LinkedIn heavily relies on artificial intelligence to deliver content and create...
- Topics:
- Hadoop,
- machine learning,
- TensorFlow,
- Data,
- Open Source