Kafka Articles

  • mock-screenshot-of-the-recruiter-usage-dashboard

    Bridging batch and stream processing for the Recruiter usage statistics dashboard

    July 14, 2020

    Co-authors: Khai Tran and Steve Weiss Batch and streaming computations are often combined together in the Lambda architecture, but carry the cost of maintaining two different code bases for the same logic. We have previously shared on the blog a behind-the-scenes look at our approach into enabling the seamless translation of declarative batch code into streaming...

  • top-blogs-logos

    The Top 2019 LinkedIn Engineering Blogs

    December 9, 2019

    As the year draws to a close, we’re taking a look back at ten of our most popular 2019 articles on the LinkedIn Engineering Blog. Examining the list, it’s clear that topics pertaining to open source and artificial intelligence are some of the most popular, as are posts that look at how we tackle technical challenges at scale. We’re excited to share new progress...

  • lag-alert-graphs

    An inside look at LinkedIn’s data pipeline monitoring system

    October 30, 2019

    Co-authors: Krishnan Raman and Joey Salacup Editor's note: This blog has been updated. Monitoring big data pipelines often equates to waiting for a long-running batch job to complete and observing the status of the execution. The status can result in “Failed” or “Successful” or even “Incomplete.” From there, it’s the team’s job to understand the impact and...

  • LinkedIn-Kafka-ecosystem

    How LinkedIn customizes Apache Kafka for 7 trillion...

    October 8, 2019

    Co-authors: Jon Lee and Wesley Wu Apache Kafka is a core part of our infrastructure at LinkedIn. It was originally developed in-house...

  • FollowFeed-flow

    Auditing content features in FollowFeed

    August 27, 2019

    The LinkedIn feed relies on a ranked list of the most relevant content for a member. More than 80% of the feed is organic content...

  • change-data-capture

    Open sourcing Brooklin: Near real-time data streaming at...

    July 16, 2019

    Editor's note: This blog has been updated. Brooklin—a distributed service for streaming data in near real-time and at scale—has been...