• dfw-architecture

    Distributed Firewall (DFW): Network security at the host level at LinkedIn

    October 21, 2021

    Co-authors: Matthew Davidson, Walter Marchuk, Andreas Zaugg, Michael Garate, and William Buenzle In traditional network design, hardware firewalls are used to provide isolation of network segments. As a network increases in size and complexity, an increasing number of firewall rules are required to provide necessary access, but the number of rules a hardware...

  • title-card

    Project Magnet, providing push-based shuffle, now available in Apache Spark 3.2

    October 20, 2021

    Co-authors: Venkata Krishnan Sowrirajan and Min Shen We are excited to announce that push-based shuffle (codenamed Project Magnet) is now available in Apache Spark as part of the 3.2 release. Since the SPIP vote on Project Magnet passed in September 2020, there has been a lot of interest in getting it into Apache Spark. As of March 2021, 100% of LinkedIn’s Spark...

  • title-card

    Our approach to building transparent and explainable AI systems

    October 7, 2021

    Co-authors: Parvez Ahammad, Kinjal Basu, Yazhou Cao, Shaunak Chatterjee, David Durfee, Sakshi Jain, Nihar Mehta, Varun Mithal, and Jilei Yang Delivering the best member and customer experiences with a focus on trust is core to everything that we do at LinkedIn. As we continue to build on our Responsible AI program that we recently outlined three months ago, a...

  • an-illustration-of-the-distributed-tier-merge

    Distributed tier merge: How LinkedIn tackles stragglers in ...

    September 27, 2021

    Co-authors: Andy Li and Hongbin Wu Indexing plays the key role in modern search engines for fast and accurate information retrieval,...

  • graph-of-linkedin-cluster-trends-for-hdfs-space-used-total-name-node-objects-and-yarn-compute-capacity

    Scaling LinkedIn's Hadoop YARN cluster beyond 10,000 nodes

    September 8, 2021

    Co-authors: Keqiu Hu, Jonathan Hung, Haibo Chen, and Sriram Rao At LinkedIn, we use Hadoop as our backbone for big data analytics and...

  •  encoded-activity-sequence-showing-requests-made-by-a-member-that-was-not-using-abusive-automation

    Using deep learning to detect abusive sequences of member...

    September 2, 2021

    Co-authors: James Verbus and Beibei Wang The Anti-Abuse AI Team at LinkedIn creates, deploys, and maintains models that detect and...