Big Data Articles

  • explodingdata1

    Managing "Exploding" Big Data

    June 15, 2017

    What is the shape of your big data? While we do love to talk about the size of our big data—terabytes, petabytes, and beyond—perhaps we are not paying due recognition to the shape of it. Big data comes in a variety of shapes. The Extract-Transform-Load (ETL) workflows are more or less stripe-shaped (left panel in the figure above) and produce an output of a...

  • async21

    Asynchronous Processing and Multithreading in Apache Samza, Part II: Experiments and Evaluation

    January 6, 2017

    This post is the second in a series discussing asynchronous processing and multithreading in Apache Samza. In the previous post, we explored the design and architecture of the new AsyncStreamTask API and the asynchronous event loop. In this post, we will focus on the study of the performance of this feature with benchmark Samza jobs. Some of the interesting...

  • Async1

    Asynchronous Processing and Multithreading in Apache Samza, Part I: Design and Architecture

    January 4, 2017

    As part of the Apache Samza 0.11 release, we rebuilt Samza’s underlying event processing engine to use an asynchronous and parallel processing model. The new model is unique among current open source stream processors because it not only supports running traditional synchronous processing in parallel on multiple threads, but also provides first-class support for...

  • streamprocess1

    Stream Processing Hard Problems Part II: Data Access

    August 22, 2016

    This post is the second in a series of posts that discuss some of the hard problems in stream processing. In the previous post, we...

  • Announcing Gobblin 0.7.0: Going Beyond Ingestion

    June 29, 2016

    About a year ago, we open sourced Gobblin, a universal data ingestion framework that aimed to solve data integration challenges faced...

  • Stream Processing Hard Problems – Part 1: Killing Lambda

    June 27, 2016

    We live in an age where we want to know relevant things happening around the world as soon as they happen; an age where digital...