Apache Kafka's popularity has grown tremendously over the past few years. In fact, LinkedIn's deployment recently surpassed 2 trillion messages per day, with over 1,800 Kafka servers (i.e., brokers). While Kafka has proven to be very stable, there are still operational challenges when running Kafka at such a scale. Brokers fail on a daily basis, which results in...
Open Source Articles
-
- Topics:
- Apache Kafka,
- monitoring,
- Kafka,
- Open Source
-
Editor’s note: Members of the information security team at LinkedIn have an opportunity to work on research topics under a well-defined framework that allows them to evaluate new products and technologies, as well as explore the related threat surface. The team strives to find new and innovative ways to help simplify and strengthen security and contribute back...
- Topics:
- Open Source,
- Security
-
Typical distributed data systems are clusters composed of a set of machines. If the dataset does not fit on a single machine, we usually shard the data into partitions, and each partition can have multiple replicas for fault tolerance. Partition management needs to ensure that replicas are distributed among machines as evenly as possible. More crucially, when a...
- Topics:
- Apache Helix,
- Big Data,
- Distributed Systems,
- Open Source
-
At a company as large as LinkedIn, service degradation isn’t a question of “if” so much as “when,” and when things do break, we need...
- Topics:
- Open Source,
- SRE
-
Co-authors: Chad Hietala and Sarah Clatterbuck Editor’s note: This is a continuation of a two-part series about the Glimmer Virtual...
- Topics:
- JavaScript,
- Ember,
- Open Source
-
In July 2016, LinkedIn unveiled Open19, a new project with the aim to establish open standards for servers based on a common form...
- Topics:
- Open19,
- 100G 50G ,
- data center,
- Open Source