In his first post, Kevin told you about a day in the life of a LinkedIn intern. In this post, he'll give you a glimpse at the end-to-end architecture of a LinkedIn app. This includes a JRuby/Sinatra front-end, Frontier back-end, and data storage in Voldemort.
LinkedIn uses a variety of technologies to process requests. Today, I'll describe one path through the site, which involves a number of new technologies that I learned while developing my first feature at LinkedIn. I'll start at the JRuby front-end (what the user sees), move to the Frontier Content Services (business logic and rules) and end up in the Voldemort data services layer (where persistent data lives). To keep things simple, I'm going to use an artificial example of a page which allows a user to store some information (website, location) for a particular company.
JRuby and Sinatra
The web app I worked on is written in JRuby and uses Sinatra for handling requests. Sinatra provides a very simple DSL for web applications, pairing HTTP methods with a pattern that matches a URL. A few simple examples:
For this post, we'll be working on an example that updates a company page with some basic information - website and location - so here is the start of the JRuby portion of the code:
For more info on how LinkedIn incorporates JRuby/Sinatra into its codebase, check out this presentation by Baq Haidri.
Passing the data to a Frontier Content Service
Once we have routed and parsed a request, the next step is to pass it through some business logic. For that, we hand off work to Frontier, a framework developed at LinkedIn to build scalable, efficient, and reliable web services. To model a particular data domain, we create a Frontier Content Service which has methods for reading and writing Content Models, type-safe Java objects representing the data the service provides. For example, a service that handles "company" data might define a CompanyContentModel as follows:
Frontier Content Services respond to the standard HTTP verbs (GET, PUT, DELETE, etc) to read and write data. Here is what the interface for the CompanyContentService, which manipulates the CompanyContentModel we defined above, might look like:
One of the most powerful features of Frontier is that communication with multiple Content Services can be done in parallel to reduce page load time. Continuing our JRuby app from before, here is how we would fetch data from two Content Services at the same time:
We created a small DSL for JRuby in which the assemble keyword processes all the Content Service calls inside it (e.g. GET resource(:profile, :v0)) in parallel. We pass parameters for each Content Service call as a Ruby hash (e.g. {"memberId" => session[:memberId]}). Since our goal is to update the company info, let's extend the JRuby app to actually call the CompanyContentService to write the new info:
Persisting the data
The next step in the process is for the Content Service to actually save these values into a persistent data store. Depending on the product requirements, we often turn to Voldemort, a distributed key-value store developed and open sourced by LinkedIn. Voldemort ensures that the data is replicated in many locations, acting as a transparent fail-safe in case some servers are inaccessible as well as offering versioning to ensure data integrity during failure scenarios.
To use Voldemort, we first configure a store (ie, "table") in stores.xml to indicate the read and write parameters that are used to maintain consistency as well as how keys and values will be serialized into bytes. In this example, I configured a store that requires 2 reads and writes and maps Integer keys (the company id) to Map<String, String> values which contain the location and website:
Now that the Voldemort store has been defined, the last step is for the Content Service to actually write to it. To do this, we pull the parameters out of the CompanyContentModel passed to us by the JRuby app, put them into a HashMap and then use the Voldemort client to store them.
Putting it all together
At the end of all of this, we have succeeded in accepting parameters from the user in the JRuby front-end, passing them through Frontier to a Java Content Service, and storing them in Voldemort. As a software engineering intern at LinkedIn, the broad range of technologies I get to use is very exciting. It seems as if every week brings a new language, framework, or program, and half the fun of my job is to learn about what they are and how to apply them to solve the problems that I am given. Things like this are the reason why I enjoy coding at LinkedIn so much, and why I look forward to coming in to the office each and every day.