monitoring Articles

  • samza-aeon-1

    Samza Aeon: Latency Insights for Asynchronous One-Way Flows

    April 19, 2018

    Co-authors: Max Wolffe and Akhilesh Gupta Introduction You can’t fix something if you don’t know there’s a problem. Measuring and tracking the latency of requests through your system is essential to identifying and resolving issues quickly. Many systems have built-in tooling which allows developers to monitor, set alerts on, and inspect the latency of a set of...

  • cruisecontrol2

    Open Sourcing Kafka Cruise Control

    August 28, 2017

    Apache Kafka's popularity has grown tremendously over the past few years. In fact, LinkedIn's deployment recently surpassed 2 trillion messages per day, with over 1,800 Kafka servers (i.e., brokers). While Kafka has proven to be very stable, there are still operational challenges when running Kafka at such a scale. Brokers fail on a daily basis, which results in...

  • ingraphs1

    InGraphs: Monitoring and Unexpected Artwork

    August 3, 2017

    At LinkedIn, we have an internal tool for visualizing operational metrics that we call inGraphs. Since I started working for LinkedIn almost four years ago, I’ve been snapshotting inGraphs that I thought were interesting—the ones that had helped to solve a problem, demonstrated a particular pattern, told an interesting story, or just plain looked cool. In...

  • inMesh visualization

    inMesh: Real-Time Monitoring of Remote Sites

    April 20, 2016

    Many IT organizations support offices distributed across the world. As the number of remote sites increases, it becomes more and more...

  • Inflow Dashboard

    InFlow - Making the LinkedIn network visible

    March 22, 2016

    To maintain the high network availability needed to serve all LinkedIn applications, we need to monitor and analyse both network...

  • Burrow: Kafka Consumer Monitoring Reinvented

    June 12, 2015

    One of the responsibilities of the Data Infrastructure SRE team is to monitor the Apache Kafka infrastructure, the core pipeline for...