Brooklin - a distributed service for streaming data in near real-time and at scale - has been running in production at LinkedIn since 2016, powering thousands of data streams and over 2 trillion messages per day. Today, we are pleased to announce the open-sourcing of Brooklin and that the source code is available in our Github repo! Why Brooklin? At LinkedIn,...
Stream Processing Articles
-
- Topics:
- scale,
- Stream Processing,
- infrastructure,
- Kafka,
- Data,
- Open Source
-
If Apache Kafka is the lifeblood of all nearline processing at LinkedIn, then Apache Samza is the beating heart pumping that blood around. Samza at LinkedIn is provided as a managed stream processing service where applications bring their logic (leveraging the wide variety of Samza APIs), while the service handles the hosting, managing, and operations of the...
- Topics:
- Stream Processing,
- Testing,
- Samza
-
The existing Lambda architecture With the evolution of big data technologies over time, two classes of computations have been developed for processing large-scale datasets: batch and streaming. Batch computation was developed for processing historical data, and batch engines, like Apache Hadoop or Apache Spark, are often designed to provide correct and complete,...
- Topics:
- Stream Processing,
- batch processing,
- Data
-
We are pleased to announce today the release of Samza 1.0, a significant milestone in the history of the project. Apache Samza is a...
- Topics:
- Stream Processing,
- Kafka,
- Open Source,
- Samza
-
A few years ago, we announced Rest.li 2.x and a Protocol Upgrade Story. Today, we are excited to share another major milestone: the...
- Topics:
- Stream Processing,
- rest.li,
- Data,
- Open Source
-
Gobblin is a distributed data integration framework that simplifies common aspects of big data integration, such as ingestion,...
- Topics:
- Stream Processing,
- Gobblin,
- Data,
- Open Source