Pinot, an open source scalable distributed columnar OLAP data store, is used extensively across almost all areas of LinkedIn. It drives member-facing use cases such as Feed Item Popularity and Who Viewed My Profile, supports recruiter tools such as Talent Intelligence, and holds the data for our internal unified metrics platform. Pinot adoption has grown dramatically in the last few years and we now have several thousand servers working continuously to serve real-time queries.
Our existing Red Hat Enterprise Linux 6 (RHEL6) installation was beginning to show its age, especially because of the limited support for the new hardware we were installing. We knew upgrading our installation would be a big undertaking—we would have to reinstall the OS from scratch—but RHEL7’s promise of some sizable performance gains won us over. In this post we’ll review how we upgraded all of our servers to RHEL7 with minimal service interruptions.
Pinot’s architecture
Pinot is architected in a simple way that makes mass migrations relatively easy and pain-free. There is a single-source-of-truth stored in a central location—such as a Network File System, Azure Data Lake, or another type of object storage. Pinot servers begin downloading any missing data segments that they are assigned, and brokers begin routing queries to the servers as soon as they finish downloading and processing segments. This means that the servers themselves are relatively easy to replace.
The migration approach
The general approach for migrating Pinot servers is simple: Spin up a new set of machines to migrate to, update the assignment policy to point to the new machines, and trigger a cluster rebalance to start moving segments around. The new machines will begin replicating and serving queries from the assigned segments. That means the old servers can be turned off to complete the migration. Brokers are even easier because they are entirely stateless. However, there are some special nuances to our upgrade that require additional care.
The upgrade to RHEL7
Pinot is a general real-time analytics database, and shines at providing up-to-date answers and statistics for a wide variety of our member-facing features. All this information is updated and accessible on-demand, which means that there are strict latency requirements to help deliver positive member experiences. Pinot provides a simple rebalancer that, for most use cases, is quick and effective, but sometimes can have a noticeable impact on performance. To secure us from these latter instances, we needed to find a different method.
Pinot is also host to a lot of our data at LinkedIn. Replicating all of this data back down from the single source of truth to all of the servers (for each of the replicas) would greatly slow down the migration. During this window of time of waiting for a new server to finish replicating its copy of the segments, the cluster is operating under reduced performance and resilience capacity. It was imperative that we found a way to shorten this time.
Instead of using the standard Pinot rebalancer, we instead devised a custom approach to migrate each cluster. We built up an identical destination cluster, and then prepared a mapping of each source server to a destination server. With these servers paired up, we performed parallel rsyncs from each source server to its matching destination to stage the data.
Diagram showing the data transfer between servers
After the data was copied, this mapping was used to transform each table’s segment assignment, replacing each source server with its matching destination server one at a time. This strategy allowed us to verify that the new servers were successfully replicating and serving traffic throughout each step of the migration.
After the new assignment was applied, Pinot picked up the staged data and began quickly serving queries from the new servers, within seconds in most cases. Some of our bigger datasets took hours to stage the data, so replicating the data ahead of time eliminated any capacity impacts during the migration.
Results
This migration provided an excellent opportunity to test and practice our server replacement procedures with all hands on deck from the team to help swap out clusters. We built in automation to schedule deployments on LinkedIn’s server management platform, and automatically picked matching hardware to swap in each cluster. All of these tools will be helpful for our regular administrative tasks, reducing the effort and the opportunities for mistakes when replacing unhealthy hardware.
As a positive side effect of our upgrade, our fleet metrics indicated a 20 to 40% reduction in CPU usage, steadily decreasing over the course of the migration, just by upgrading from RHEL6 to RHEL7.
Graph showing average CPU usage over time
Conclusion
Overall, Pinot successfully completed the migration to RHEL7 on time while keeping the site up and running at full capacity. By upgrading to a newer OS, we gained several tangible benefits:
- We reduced our CPU usage by 20 to 40%, paving the path towards further hardware cost savings.
- The newer Linux kernel brings better SSD performance.
- We greatly improved Pinot’s automation on top of LinkedIn’s deployment platform.
This migration shows the benefits of investing the effort for an upgrade. Now, the LinkedIn Pinot team is also much better equipped for future hardware and OS migrations.
Acknowledgments
A lot of people pulled together to make this happen successfully. On the LinkedIn platform side, Jamie Henderson, Rohit Jamuar, and Sumit Sanghrajka were key to the success of this project. I’d like to thank Dino Occhialini, John Gutmann, and all the Pinot SREs and devs for providing useful feedback and helping execute the migration plan. Thanks to the leadership team, Prasanna Ravi, Nirupam Mohari, and Rishi Pande and for their continued support during this migration.