Debrief from Black Hat - The Tactical Application Security Program

Getting Stuff Done

August 7, 2015

This week, members of LinkedIn’s security team descended on Las Vegas to participate in Black Hat. My colleague Cory and I presented a talk on building a solid tactical security program. We’ve offered some of the key insights here, for those unable to attend, as well as some questions we got afterward, at the bottom of this post.

How To Be Tactical
At LinkedIn, we consider our security program to be tactical first, and we’re dedicated to this approach. Application Security (AppSec) is one of the most dynamic disciplines in the field - if a team’s focus remains only on building out a long term strategy and executing upon it, you will fail at your mission of securing the business. Tactical approaches return value immediately and can adapt. The key elements of our tactical application security program are:

  1. Adopt lightweight approaches to assessment and response, and iterate on them frequently.

  2. Constantly ask ourselves tough questions about the work we’re doing to make sure we’re demonstrably addressing risk or vulnerability. (If we don’t like the answers, we change the plan now rather than later.)

  3. Consider operational excellence to be our primary goal,and collect key performance indicators to measure how we’re doing.

  4. Favor plans that make people move rather than wait. Executing against a small set of iterative improvements that involve other teams and stakeholders allows us to keep focus on the current problems at hand.

This tactical program is not without an overall strategy of programmatic success - we’ve found that when you execute successfully against a tactical plan, people recognize that this is a strategic approach in itself.

Be Consultative
Running a security organization that involves reviewing products and technologies should be run like a consultancy. Core in this is defining a set of operating principles; at LinkedIn, we are guided by the following:

  1. We should be easy-to-find and engage – it should feel like you’re speaking with an advisor who’s always available to give you guidance.

  2. We should be conscious of our colleagues’ time, and be flexible on documentation and preparation – as long as we have information to determine next steps, we should strive to be agile.

  3. Our guidance should be clear, understandable, and have a reasonable likelihood of execution.

  4. We should assist our customers to arrive at solutions, including gathering guidance from legal, customer support, engineering and others.

  5. We should be passionate yet practical – working with us should not feel like judgment or punishment.

Response is Key
The effectiveness and reputation of a security organization is most frequently judged by how they respond to incidents and security bugs. If you do things well, people will rarely see your assessment program and how it works. Doing AppSec incident response though, is more than just quickly handling vulnerabilities that pop up.

Core to our approach is a lightweight playbook which provides a framework to guide our team through the response process. This is not a 100 page document that no one ever reads or updates; it’s practical and predictable, with a framework for classifying bugs and the teams to involve in resolution.

Another piece that many teams miss is communication. Providing a centralized source of information for all individuals involved requires coordination. Communication must also flow to others in the organization outside of the incident, such as management, PR and sometimes legal. Keep this communication consistent, direct and upbeat.

We have published examples of both these playbook elements and the full slides of our presentation at;;

Considering a Bounty Programs? Focus on Relationships
Before considering any type of bug bounty program, you first have to consider the parameters  of handling external vulnerability reports. This includes building templates for responses, establishing internal SLAs for responding, building agreements with engineering and dev teams on SLAs for resolution, and a clearly defined table of bugs with their related severity.

We created our program with researchers in mind – we appreciate working with people dedicated to coordinated disclosure practices, and engage them in a deeper and mutually rewarding relationship. Our suggestion: start small with a group of people you trust.

Additional thoughts on our approach and background on our private bug bounty program can be found in our announcement post by Cory.

Closing Notes
Big thanks to all of the attendees who came to listen to our talk. Afterward, we received some great questions from the audience including how to scale our tactical approach to companies both smaller and larger than LinkedIn.

With smaller companies, the key is to generate demand for your services. Executing well and demonstrating success will generate demand, allowing you to add resources (staff, tooling, capabilities) with greater ease.

Larger organizations present a unique issue of having to cover significant ground without doubling or tripling their security team. Focus on a targeted approach, choosing parts of the organization you believe present the largest risk. Wins in these targeted areas will allow you to move laterally within the organization and continue to increase your foothold.

Having an approach that is lightweight, tactical, measurable and adaptable is key to a successful program in organizations of any size.

Lastly, I would like to say thank you to Black Hat and the Review Board. It was a great opportunity to share our thoughts and opinions to the larger security community.