Project Voldemort 0.96 Release

I am happy to announce the availability of the Voldemort 0.96 open source release.

What's happening

We really appreciate the support, patience, and valuable feedback from the open source community and our users. Since the last Voldemort open source release, the Voldemort team at LinkedIn has been actively working on improving the manageability and the operability of Voldemort clusters. In particular, in order to keep up with Voldemort’s high adoption rate at LinkedIn, one of the primary goals is to be able to operate multi-tenant clusters, thus reducing the total number of clusters that we need to manage.

To simplify the resource management in a multi-tenant Voldemort cluster and to provide faster response time, the Voldemort team has been tuning Voldemort servers, especially the BDB JE storage engine, to run on Solid State Drives. However, it has been a long journey for the team to tune the system to run on SSD, mainly due to the large scale deployment of Voldemort at LinkedIn, the intricacy of the memory management of the JVM based systems, and the subtle issues encountered in the newer versions of BDB JE (4.1.17 and 5). More details on the tuning work can be found in a separate blog post we published here.

Even though our work on performance tuning Voldemort for SSDs continues, we are doing a release now to share our initial improvements and to share other significant enhancements we have made.

About this release

In addition to many small features, enhancements, and bug fixes, this release includes significant improvements in Rebalancing, Read-only Pipeline, and performance monitoring for both client and server (including extensive stats on the BDB storage engine) through mbeans. Please refer to the release notes for more details, including an overview on the new Rebalancing work and Read-only pipeline data fetch optimization.

Next play

Although the Voldemort open source community has been quiet, the Voldemort team at LinkedIn had been actively working on Voldemort to make it better. Our current development roadmap is centered around how to build and operate Voldemort as a Service, namely VaaS. The roadmap includes the resource management (I/O, memory, network bandwidth and connections, etc), metadata management, GUI-based management console (currently with some LinkedIn specific components), performance lab infrastructure, and source-agnostic streaming-in platform. Those features will be made available to our users through a few open source releases in the next 6 – 12 months.

We also need some feedback from the open source community on how useful it would be to have a consistent event change log from a Voldemort cluster, similar to the redo log of an Oracle instance or the MySql bin log, though without the transactional semantics. The consistent event change log could enable streaming data between Voldemort clusters as well as from a Voldemort cluster to other data sources.

A note to the open source community: help us make Voldemort better!

The Voldemort team values the open source community. We love to hear and discuss suggestions on how to improve Voldemort to run faster or be more capable; we work hard to quickly fix bugs found by our open source users; and we want to help open source contributors become more involved in our development process.

To consolidate the open source issue tracking with the actual source code, we are cleaning up and moving tickets from google code to github.

To wrap up this blog post, I’d like to ask the open source community the following questions. You can either directly comment on the blog post or post on the project forum.

1. Besides the features/enhancements mentioned in “Next play” section, what is the one feature/enhancement you like to see in Voldemort?
2. How often would you like to see the development updates from the Voldemort team – for features/enhancements initiated by the team at LinkedIn?
3. How could we improve the communication between the Voldemort team at LinkedIn and the open source community?