Roshan Sumbaly, Jay Kreps, Alex Feinberg, Lei Gao, and Sam Shah

In the 10th USENIX conference on File and Storage Technologies (FAST 2012)




Current serving systems lack the ability to bulk load massive immutable data sets without affecting serving performance. The performance degradation is largely due to index creation and modification as CPU and memory resources are shared with request serving. We have extended Project Voldemort, a general-purpose distributed storage and serving system inspired by Amazon’s Dynamo, to support bulk loading terabytes of read-only data. This extension constructs the index offline, by leveraging the fault tolerance and parallelism of Hadoop. Compared to MySQL, our compact storage format and data deployment pipeline scales to twice the request throughput while maintaining sub 5 ms median latency. At LinkedIn, the largest professional social network, this system has been running in production for more than 2 years and serves many of the data-intensive social features on the site.



@inproceedings{Sumbaly:2012:SLB:2208461.2208479, author = {Sumbaly, Roshan and Kreps, Jay and Gao, Lei and Feinberg, Alex and Soman, Chinmay and Shah, Sam}, title = {Serving large-scale batch computed data with {Project Voldemort}}, booktitle = {Proceedings of the 10th USENIX conference on File and Storage Technologies}, series = {FAST'12}, year = {2012}, location = {San Jose, CA}, numpages = {13}, url = {}, acmid = {2208479}, publisher = {USENIX Association}, address = {Berkeley, CA, USA}, }