Open Source Articles

  • migz1

    MiGz for Compression and Decompression

    February 20, 2019

    Compressing and decompressing files with GZip normally uses a single thread. For large files, this can bottleneck dependent tasks like data processing, data analysis, and machine learning. Although there are several alternatives supporting multithreaded compression, such as pigz (command-line tool) and ParallelGZip (Java library), no Java library (for any...

  • ccfe1

    Introducing Kafka Cruise Control Frontend

    February 7, 2019

    At LinkedIn, Kafka is the de-facto messaging platform that powers diverse sets of geographically-distributed applications at scale. Examples include our distributed NoSQL store (Espresso), stream processing framework (Samza), monitoring infrastructure (InGraphs), and derived data serving platform (Venice). Given these use cases, it’s not surprising that Kafka...

  • helixtask1

    Managing Distributed Tasks with Helix Task Framework

    January 24, 2019

    Co-authors: Junkai Xue and Hunter Lee   Stateless tasks are widely used for serving large-scale data processing systems. Lots of requests were made by systems, which rely on Apache Helix, for a stateless task management feature to be added to Apache Helix. Recently, our team decided to explore new ways to manage stateless tasks, in addition to our ongoing work...

  • samzalogo

    Samza 1.0: Stream Processing at Massive Scale

    November 27, 2018

    We are pleased to announce today the release of Samza 1.0, a significant milestone in the history of the project. Apache Samza is a...

  • unstructureddata1

    Unstructured Data Transfer in Rest.li

    November 2, 2018

    A few years ago, we announced Rest.li 2.x and a Protocol Upgrade Story. Today, we are excited to share another major milestone: the...

  • selffocused1

    Making Ember Applications' UI Transitions Screen Reader...

    October 17, 2018

    Note: An image of the LinkedIn-created logo for the self-focused open source project accompanies this post.   At LinkedIn, we strive...