Gobblin is a distributed data integration framework that simplifies common aspects of big data integration, such as ingestion, replication, organization, and lifecycle management, for both streaming and batch ecosystems. Gobblin has been gobbling big data with ease in the open source world since December 2014. Over the years, Gobblin has evolved at a tremendous...
Stream Processing Articles
-
- Topics:
- Stream Processing,
- Big Data,
- Gobblin,
- Open Source
-
Over the last two years at LinkedIn, I’ve been working on a distributed key-value database called “Venice.” Venice is designed to be a significant improvement to Voldemort Read-Only for serving derived data. In late 2016, Venice started serving production traffic for batch use cases that were very similar to the existing uses of Voldemort Read-Only. In the time...
- Topics:
- Stream Processing,
- Big Data,
- Kafka,
- serving infrastructure
-
Co-authors: Saurabh Goyal and Janardh Bantupalli In our previous blog post introducing Brooklin, we outlined the reasons why we created our own framework for near real-time incremental data capture from production. This framework feeds data to our larger data ingestion pipeline for the hundreds of nearline applications processing data that are distributed across...
- Topics:
- Stream Processing,
- database,
- Distributed Systems
-
Near-realtime (nearline) applications drive many of the critical services within LinkedIn, such as notifications, ad targeting, etc....
- Topics:
- Stream Processing
-
Over a decade ago, test strategies invested heavily in UI-driven tests. Backend and mid-tier services were tested using automated...
- Topics:
- Apache Samza,
- Stream Processing,
- Apache Kafka,
- Testing
-
This post is the second in a series discussing asynchronous processing and multithreading in Apache Samza. In the previous post, we...
- Topics:
- Apache Samza,
- Stream Processing,
- Apache Kafka,
- Big Data