Co-authors: Jon Lee and Wesley Wu Apache Kafka is a core part of our infrastructure at LinkedIn. It was originally developed in-house as a stream processing platform and was subsequently open sourced, with a large external adoption rate today. While many other companies and projects leverage Kafka, few—if any—do so at LinkedIn’s scale. Kafka is used extensively...
scale Articles
-
- Topics:
- scale,
- Kafka,
- Open Source
-
Introduction LinkedIn is committed to providing economic opportunities for every member of the global workforce, and we’re growing at a rapid pace. Our platform is built on a collection of large-scale multi-cluster services functioning in harmony to offer a unified product experience to members. The aim of several backend engineering teams is to allow other...
- Topics:
- scale,
- infrastructure
-
Co-authors: Yu Li, Szymon Gizecki, and Chinmaya Dattathri To ensure we have significant flexibility in how our teams collaborate, our trunk-based engineering development workflow manages dependencies on a binary level, instead of source level. This requires very efficient and sophisticated management of the resulting dependency graph, and we discussed our...
-
The pursuit of our mission to connect the world’s professionals to make them more productive and successful is deeply dependent on the...
- Topics:
- scale,
- infrastructure
-
Brooklin - a distributed service for streaming data in near real-time and at scale - has been running in production at LinkedIn since...
- Topics:
- scale,
- Stream Processing,
- infrastructure,
- Kafka,
- Open Source
-
Pinot, a scalable distributed columnar OLAP data store developed at LinkedIn, delivers real-time analytics for site-facing use cases...
- Topics:
- scale,
- Developer Productivity,
- Pinot,
- infrastructure