Enhancing Security and Developer Productivity: LinkedIn's Journey with Implementing Content Security Policy
Co-authors: Covenant Goo, Matthew Lemons, Mira Thambireddy, and Roman Shafigullin
LinkedIn Information Security is committed to help foster a community that is safe and secure for our members. The Application Security team is responsible for safeguarding LinkedIn member data through the implementation and management of various security features, focusing primarily on framework-level security. One of our core responsibilities at the web framework layer is to configure and manage security headers to enhance web application security, some of which include Content Security Policy (CSP), HTTP Strict Transport Security (HSTS), and Cross-Origin Resource Sharing (CORS).
In this post, we'll discuss our journey in adopting and maintaining Content Security Policy. Our focus was on improving developer productivity through security automation, eliminating the need for manual intervention by the Application Security team. We'll also explore the lessons we learned from implementing two different architectural approaches - a simple rule-intensive centralized model and a developer-empowered decentralized approach.
Our legacy CSP architecture was a centralized model, where outgoing HTTP responses were matched against a set of rules to get the appropriate set of CSP values assigned. Our team maintained these collection of rules and deployed them to LinkedIn proxy hosts where a plugin would read the CSP configs and monitor outgoing responses. We built a list of rules for each path or groups of paths or applications and applied the rules through appropriate matching to outgoing responses.
Figure 1. Centralized system
The Traffic Headers Plugin wrote CSP headers into responses before sending the responses back to the client. Changes to code and rules at the traffic layer are risky and result in slower progress as this is our infrastructure that touches every request and response with LinkedIn.
Overall, with a centralized approach, we initially saw tremendous benefits given the finite number of application configs to maintain. We possess the ability to make changes and update rules in a single location. We gain better readability of rules given that configs are in an easier human-readable format. Another notable benefit of this approach is the assurance of consistent implementation of secure CSP policies given that the security team is involved in making changes.
This simple and secure approach was effective for a limited set of applications. As we grew, our list of rules grew, the order of the list became important, and the number of CSP versions increased to over 1,000. We came to a realization that this approach lacked scalability. From a process standpoint, having a centralized architecture meant that the Application Security team was in the critical path to review and approve CSP changes for content updates on LinkedIn. A human error in the configurations could leave a large vulnerability or even cause site-up issues if our application that sets CSP headers fails to load and blocks all outgoing responses. From a developer perspective, testing if new page content works with CSP was difficult due to the process we mentioned above and the location that we set CSP headers, mainly since we set in a centralized place a developer would not be able to do the same for a test run on their local machine.
The Application Security team started to do some research into ways that could improve our current architecture and solve for some of the issues we identified. We adopted a decentralized approach after a bit of research into the newest CSP standards and LinkedIn infrastructure, as well as discussing possible options ranging from an overhaul of the current system to scrapping it completely and starting fresh.
We would keep the old centralized architecture around for legacy applications and as a fallback point but would migrate as many applications as possible to our new decentralized system. We had five main goals of this new system:
Minimize the blast radius of changes
Empower developers to make changes
Provide a consistent testing experience
Improve LinkedIn’s security posture
Build a generalized solution to support more than just CSP headers
Figure 2. Decentralized system
In our effort to decentralize Content Security Policy headers, we created a CSP Filter, which is an intercepting filter. This filter lives as part of our frontend frameworks.
Developers can define Content Security Policies as part of their app’s configurations. When a request comes in, the WebApp processes the request and produces a response. As the response is leaving the WebApp, the response is intercepted by our new CSP Filter. If the WebApp owner has defined CSPs as part of their configurations, then the CSP Filter will decorate the response with Content Security Policy response headers. If the WebApp owners have not defined CSPs as part of their configurations, then the CSP Filter will take no action.
After a response leaves the WebApp layer and returns to the traffic layer, our existing Traffic Headers Plugin looks at the response. If there are no Content Security Policy headers in the response, the plugin will decorate the response with the appropriate headers. This fallback mechanism ensures setting a CSP header for every response.
Overall with the decentralized system, we saw a reduced impact in CSP changes, ensuring that changes have a limited and well-contained scope within the application thus minimizing the potential errors and disruptions. We were able to equip developers with the necessary tools, resources and knowledge to efficiently implement by themselves thereby empowering developer productivity. Additionally, developers could leverage their own testing environment for testing CSP changes.
One primary disadvantage in adopting a decentralized architecture for Content Security Policy customizations is that the security governance for CSP rules became challenging due to the following reasons:
Lack of centralized location to easily inspect CSP rules.
While developers are empowered to make their own changes, the Application Security team is not directly involved in setting CSP rules leading to lack of visibility into any potentially unsafe CSP changes.
Modification of CSP can be tricky for developers and sometimes requires deep knowledge of frontend security.
We addressed these issues by adopting a shift-left approach to implement security validators that analyze code at the time of code commit. To ensure developers always commit safe CSP policies, we implemented risk-based validation rules. We leveraged GitHub validation checks to enforce the rules and appropriate actions for the developer to take using GitHub annotations.
As an example, one critical risk rule we defined is that we support only static JavaScript using hashes or loaded from trusted sources. This means that we block a developer’s pull request (PR) when they set script-src directive to a wildcard or a domain that is not approved. The following is a screenshot of the developer experience for critical rules.
At LinkedIn, our commitment to delivering a safe and trustworthy platform for our members led us on a transformative journey that not only bolstered our security posture but also empowered our developers. Today, we have successfully transitioned towards a decentralized model that effectively empowers our developers and enhances our overall security posture. As we continue on this journey, we look forward to further advancements, always striving to provide an unparalleled experience for our members while maintaining the highest security standards.