Compressing and decompressing files with GZip normally uses a single thread. For large files, this can bottleneck dependent tasks like data processing, data analysis, and machine learning. Although there are several alternatives supporting multithreaded compression, such as pigz (command-line tool) and ParallelGZip (Java library), no Java library (for any...
Open Source Articles
-
- Topics:
- Open Source
-
At LinkedIn, Kafka is the de-facto messaging platform that powers diverse sets of geographically-distributed applications at scale. Examples include our distributed NoSQL store (Espresso), stream processing framework (Samza), monitoring infrastructure (InGraphs), and derived data serving platform (Venice). Given these use cases, it’s not surprising that Kafka...
- Topics:
- Apache Kafka,
- Kafka,
- Open Source
-
Co-authors: Junkai Xue and Hunter Lee Stateless tasks are widely used for serving large-scale data processing systems. Lots of requests were made by systems, which rely on Apache Helix, for a stateless task management feature to be added to Apache Helix. Recently, our team decided to explore new ways to manage stateless tasks, in addition to our ongoing work...
- Topics:
- Apache Helix,
- Distributed Systems,
- Open Source
-
We are pleased to announce today the release of Samza 1.0, a significant milestone in the history of the project. Apache Samza is a...
- Topics:
- Stream Processing,
- Kafka,
- Open Source,
- Samza
-
A few years ago, we announced Rest.li 2.x and a Protocol Upgrade Story. Today, we are excited to share another major milestone: the...
- Topics:
- Stream Processing,
- rest.li,
- Open Source
-
Note: An image of the LinkedIn-created logo for the self-focused open source project accompanies this post. At LinkedIn, we strive...
- Topics:
- Accessibility,
- Ember,
- Open Source