Kartik Paramasivam

Posts by Kartik Paramasivam

  • streamprocess1

    Stream Processing Hard Problems Part II: Data Access

    August 22, 2016

    This post is the second in a series of posts that discuss some of the hard problems in stream processing. In the previous post, we explored the use of lambda architecture in stream processing and discussed techniques to avoid it. In this post, we’ll focus on one of the main bottlenecks in high scale stream processing applications: “accessing data.” Background...

  • Stream Processing Hard Problems – Part 1: Killing Lambda

    June 27, 2016

    We live in an age where we want to know relevant things happening around the world as soon as they happen; an age where digital content is updated instantly based on our likes and dislikes; an age where credit card fraud, security breaches, device malfunctions and site outages need to be detected and remedied as soon as they happen. It is an age where events are...

  • RocksDB

    Stream processing with Apache Samza - Current and Future

    January 26, 2016

    At LinkedIn, events pertaining to application and system monitoring, member behavior tracking, inter-application communication, etc., are all ingested into our pub-sub messaging system (Kafka). A staggering 1.3 trillion events are published into Kafka per day with peaks of 4.5 million messages/sec per cluster (these numbers don’t correlate directly with site...

  • How We’re Improving and Advancing Kafka at LinkedIn

    September 2, 2015

    Kafka continues to be one of the key pillars in LinkedIn’s data infrastructure. One of our engineers has described it as LinkedIn’s...