Co-authors: Min Shen, Chandni Singh, Ye Zhou, and Sunitha Beeram At LinkedIn, we rely heavily on offline data analytics for data-driven decision making. Over the years, Apache Spark has become the primary compute engine at LinkedIn to satisfy such data needs. With its unique features, Spark empowers many business-critical tasks at LinkedIn, including data...
-
- Topics:
- Spark,
- infrastructure,
- Data,
- Open Source
-
As companies grow, adapt, morph, and mature, one item remains the same: the need for reinvention. Technical infrastructure is no exception. As our member community grew, our priorities were to keep up with that growth, or as we say, ensure continuous “site up.” (Read: adding servers to scale from hundreds to hundreds of thousands.) We ran into challenges about...
- Topics:
- Performance,
- infrastructure,
- linux,
- SRE
-
Co-authors: Benjamin Le, Daniel Gmach, Aman Grover, Roshan Lal, Jerry Lin, Austin Lu, Qingyun Wan, and Leighton Zhang Feature engineering is foundational for building artificial intelligence (AI) that powers products at LinkedIn. Recently, “representation learning” or “feature learning” has started replacing manually engineered features, as they provide...
-
Co-authors: Parag Agrawal, Ankan Saha, Yafei Wang, Aastha Nigam, and Eric Lawrence Figure 1: A heterogeneous social network LinkedIn’s...
-
Co-authors: Yixing Zhang, Bingfeng Xia, Ke Wu, and Xinyu Liu Since Beam Samza runner was developed in 2018 at LinkedIn, we now have...
- Topics:
- Stream Processing,
- Apache Samza,
- Performance,
- Benchmark
-
Last week, the global LinkedIn Data Science team joined together for our third-annual Data Science Week. This virtual event included...
- Topics:
- data science,
- engineering culture