A Search Trip Through Time

At a recent invitation-only gathering about Search technology hosted by LinkedIn’s Bangalore office, Igor Perisic, VP Engineering at LinkedIn, took us on a trip down memory lane. Igor shared the state of LinkedIn’s search technology at its inception in the early 2000’s and progressed to how we reached the current scale of having 364 million registered members across the globe hit our search index regularly. Core to this talk was the idea of the evolution of our search technology, or as Igor said, “At each iteration, we did what we thought was the smartest thing at the time.”

If you don't work in this field, it's worth stating that search comes in many varieties. Web search (Google), product search (Amazon), vertical search (querying your company intranet), and people search (probably Facebook), to name just a few. Each of those require technology and information retrieval algorithms specific to that search type. Roughly speaking, web search is driven largely by the content being indexed and the query itself. Product and vertical search also depend on additional side information like taxonomies and some structured information that’s more easily available there. People search adds the social graph as a core component.

Search at LinkedIn is a combination of all of the above. It needs to be sensitive to the social graph (e.g., “in your network”), structured information in a profile (“works at Microsoft”), and also the text content (“exceptional delivery manager”). Add to that the need to scale to big data and high query rates and you have a complex beast.

What was most striking to me, as Igor took us through this temporal journey from 2003 to 2015, was how two elements remained in motion simultaneously. First, the technology moved from a database and an early-edition Lucene index and social graph representation – all your modern-in-2003 goodness – to a full-blown Search-as-a-Service solution – all your currently modern search engine goodness. Second, the technology choice was always shaped by feature requirements (like typeahead, faceted search, separating people closely connected to you from those far away in the social graph) and the need to scale (from practically zero in 2003 to 364 million members in 2015).