Open Source Hits Scale in 2015

December 22, 2015

Open Source

In 2015 we made our biggest contributions yet to the open source community by open sourcing more than 10 original projects, including Pinot, Burrow and Gobblin, and pushing significant updates to Samza,, Kafka and Voldemort, four of our most broadly adopted open source projects.

We‘ve worked to scale our infrastructure as we reached 400 million LinkedIn members, so it’s no surprise many of our open source projects this year focus on building out our data pipelines and tools to help make sense of our data. The infrastructure improvements we’ve made in Kafka have allowed us to handle 1.3 trillion messages per day, and Espresso now serves 2.2 million rows per second. This trend of scaling, of course, runs parallel with some of the larger trends among tech companies. Producing and acting on insights built from data is a foundational element of today’s technology companies, and having the correct data is critical to getting those insights. There are dozens and dozens of complex issues around this single problem: sourcing the correct data, getting it into a data warehouse, using the right tools and schemas to sort it and create those insights.

Open source is no longer just a community or hobby for engineers, it’s part of core business strategy that we are fully invested in and something that will produce projects that will become – or have become – key pieces of LinkedIn’s infrastructure. This strategy helps push our engineers to write better software, develop faster, and become part of a larger community of many of the world’s most talented engineers. While 2015 was a great year for our open source contributions, we have a strong pipeline of projects we plan to open source in 2016 that we believe will be valuable for companies to adopt.

Here’s a look at some of the highlights from LinkedIn’s year in open source.

2015 Milestones

This data at scale revolution is only getting started, of course, and the future holds a lot of really exciting work for data-focused open source projects, both from LinkedIn and the larger community. In the coming years, I think we’ll see great advances at the streaming data layer, which hasn't yet reached maturity, as well as a continued focus on cloud computing through the entire stack. Whether it is on the container level (Docker, etc.) or the platform level (Hadoop, Spark, etc.)

It will be interesting to track the impact open source will make on more consumer-facing industries like drone technology and the Internet of Things. It is bound to make a massive impact on healthcare – the explosion of digital health information can only work correctly on a common platform, and that will most likely be open source. Open source is already affecting every single domain that traffics in code, and that impact will only continue to grow.