Co-authors: Andy Li and Hongbin Wu Indexing plays the key role in modern search engines for fast and accurate information retrieval, and the ability to swiftly build indexes is crucial for LinkedIn to provide up to date information, such as candidates to recruiters, job posts to members, etc. In some instances, such as if a member profile is missing and...
Data Articles
-
- Topics:
- Spark,
- Data,
- Distributed Systems
-
Co-authors: Keqiu Hu, Jonathan Hung, Haibo Chen, and Sriram Rao At LinkedIn, we use Hadoop as our backbone for big data analytics and machine learning. With an exponentially growing data volume, and the company heavily investing in machine learning and data science, we have been doubling our cluster size year over year to match the compute workload growth. Our...
- Topics:
- Hadoop,
- Data,
- Open Source
-
Co-authors: Ze Mao, Matt Wise, Casey Getz, Justin Lin, Ashish Singhai, and Rob Block Introduction Ambry is LinkedIn's scalable geo-distributed object store. Developed in-house and open sourced in 2016, Ambry stores tens of petabytes of data. At LinkedIn, Ambry is used to store objects like photos, videos, and resume uploads, as well as internal binary data....
- Topics:
- infrastructure,
- Data,
- Storage,
- Open Source
-
Co-authors: Kirill Talanine, Jeffrey D. Gee, Rohan Ramanath, Konstantin Salomatin, Gungor Polatkan, Onkar Dalal, and Deepak Kumar...
-
Co-authors: Xiaoyang Gu, Xie Lu, and Xiaoguang Wang Introduction In August 2019, we introduced our members and customers to the idea...
- Topics:
- infrastructure,
- scale,
- Data
-
Within a matter of hours of each day beginning, we ingest tens of billions of records from online sources to HDFS, aggregated across...