Stream Processing Articles

  • brooklin-1

    Streaming Data Pipelines with Brooklin

    October 11, 2017

    Near-realtime (nearline) applications drive many of the critical services within LinkedIn, such as notifications, ad targeting, etc. These applications need continuous and low-latency access to data, but moving data quickly and reliably from the storage layer to the nearline applications is a non-trivial problem. Building one-off data pipelines that serve the...

  • testing_samza1

    Test Strategy for Samza/Kafka Services

    April 27, 2017

    Over a decade ago, test strategies invested heavily in UI-driven tests. Backend and mid-tier services were tested using automated UI-based tests. While UI-based tests have certain merits, such as testing user flows, they are also time-consuming and fragile. The strong coupling of tests with UI caused several problems: Tests needed frequent modification due to...

  • async21

    Asynchronous Processing and Multithreading in Apache Samza, Part II: Experiments and Evaluation

    January 6, 2017

    This post is the second in a series discussing asynchronous processing and multithreading in Apache Samza. In the previous post, we explored the design and architecture of the new AsyncStreamTask API and the asynchronous event loop. In this post, we will focus on the study of the performance of this feature with benchmark Samza jobs. Some of the interesting...

  • Async1

    Asynchronous Processing and Multithreading in Apache Samza,...

    January 4, 2017

    As part of the Apache Samza 0.11 release, we rebuilt Samza’s underlying event processing engine to use an asynchronous and parallel...

  • streamprocess1

    Stream Processing Hard Problems Part II: Data Access

    August 22, 2016

    This post is the second in a series of posts that discuss some of the hard problems in stream processing. In the previous post, we...

  • Stream Processing Hard Problems – Part 1: Killing Lambda

    June 27, 2016

    We live in an age where we want to know relevant things happening around the world as soon as they happen; an age where digital...