Co-authors: Matthew Davidson, Walter Marchuk, Andreas Zaugg, Michael Garate, and William Buenzle In traditional network design, hardware firewalls are used to provide isolation of network segments. As a network increases in size and complexity, an increasing number of firewall rules are required to provide necessary access, but the number of rules a hardware...
-
- Topics:
- Security,
- trust engineering
-
Co-authors: Venkata Krishnan Sowrirajan and Min Shen We are excited to announce that push-based shuffle (codenamed Project Magnet) is now available in Apache Spark as part of the 3.2 release. Since the SPIP vote on Project Magnet passed in September 2020, there has been a lot of interest in getting it into Apache Spark. As of March 2021, 100% of LinkedIn’s Spark...
- Topics:
- Spark,
- Open Source
-
Co-authors: Parvez Ahammad, Kinjal Basu, Yazhou Cao, Shaunak Chatterjee, David Durfee, Sakshi Jain, Nihar Mehta, Varun Mithal, and Jilei Yang Delivering the best member and customer experiences with a focus on trust is core to everything that we do at LinkedIn. As we continue to build on our Responsible AI program that we recently outlined three months ago, a...
-
Co-authors: Andy Li and Hongbin Wu Indexing plays the key role in modern search engines for fast and accurate information retrieval,...
- Topics:
- Spark,
- Data,
- Distributed Systems
-
Co-authors: Keqiu Hu, Jonathan Hung, Haibo Chen, and Sriram Rao At LinkedIn, we use Hadoop as our backbone for big data analytics and...
- Topics:
- Hadoop,
- Data,
- Open Source
-
Co-authors: James Verbus and Beibei Wang The Anti-Abuse AI Team at LinkedIn creates, deploys, and maintains models that detect and...