Kafka
Data Infrastructure
Apache Kafka is a core part of our infrastructure at LinkedIn. It was originally developed in-house as a stream processing platform and was subsequently open sourced. Today, it’s widely used by the industry, has an active community, and few companies — if any — do so at LinkedIn’s scale.
Team & Project Spotlights
Cruise Control
At LinkedIn, Cruise Control handles the large-scale operational challenges of running Apache Kafka. We use it to maintain our clusters healthy in a proactive and automatic way by balancing, reacting, and tuning. Cruise Control was open sourced a few years ago and has been thriving outside LinkedIn.
Kafka Ecosystem at LinkedIn
Kafka is used extensively throughout our software stack, powering use cases like activity tracking, message exchanges, metric gathering, and more. We maintain over 100 Kafka clusters with more than 4,000 brokers, which serve more than 100,000 topics and 7 million partitions, resulting in over 7 trillion messages processed daily.
Interested in joining our team at LinkedIn?