Security metric techniques: How to answer the 'so what?'

You need to be ready when the boss responds to your presentation with a "so what?" At Metricon 5, the focus is on several security metric techniques to pull it off.

WASHINGTON, D.C. -- For the last five years, the people behind Securitymetrics.org have held their annual Metricon event to contemplate new ways of measuring risk and best communicating them to executives. This year, the security metric techniques being discussed revolve around the art of language.

Andrew Jaquith, senior analyst with Forrester Research and host of this year's event, said the risks are crystal clear. Referencing a "rolling snapshot" WhiteHat Security Founder and CTO Jeremiah Grossman conducted between January 2006 and August 2007, Jaquith noted that in that timeframe, seven out of 10 websites from the 128 million scanned had critical or urgent vulnerabilities. The issue at hand is how to put those vulnerabilities and the damage they can cause into the proper perspective for the CEO or board of directors.

Also see "A few good security metrics"

Following Jaquith to the podium with some of the answers was Richard Seiersen, security principal at Kaiser Permanente, one of the world's largest healthcare organizations. His job is to keep the massive pile of medical records and other patient information from getting stolen through system vulnerabilities attackers try to exploit.

One of his main messages was that security practitioners must be out in front of the inevitable question executives will ask after being told the company has vulnerabilities that must be fixed with additional investments in technology and people: "So what?"

"The first question you'll get is 'so what?'"Seiersen said. "They want you to tell them 'why this information is important to me?'"

His approach is to present security metrics in the "fourth dimension." There are three standard dimensions metrics are based on, he said: value, time and risk. To get beyond the "so what" question the practitioner must be able to offer clear examples of not just what and where the risks are, but what kinds of valuable business resources are threatened, which in turn will help executives understand the value in fixing them. Time is about when something needs to be fixed by and why.

He said the next question that will be asked is "What are you doing about the problem?"

Enter the fourth dimension of security metrics: Effectiveness.

This is where language comes in. Seiersen cautioned practitioners to never use language like this: "Out-of-cycle remediation should decrease & there should be high correlation with exploitability and risk, etc."

A better way to put it is something like this: "These actively exploitable flaws [threaten] Internet access and our critical business applications and the solution must be deployed in one business day by the end of the fourth fiscal quarter." Putting it in those words is more direct and makes it clear why certain investments may be needed and, once purchased, deployed quickly.

When it comes to measuring the success rate of actions taken and where the organization needs to go from there, Seiersen suggested a graphic that captures the four dimensions and includes green, yellow and red to demonstrate progress and failure. In an example, he showed a chart where the overall picture is that five areas of focus are in the green, 50 are in the yellow and 147 is in the red. That's a high failure rate and must be followed up with a description of why it's bad and what needs to be done in a finite period of time.

In the final analysis, the best metrics and actions won't reduce 100 percent of the risk. But looking at it in four dimensions will lower the risk considerably, he said.

"For us it's straightforward: No patient data gets out, even if it means spending $10 million," he said.

The key to getting that money is to get past the question of "so what?"

Join the discussion
Be the first to comment on this article. Our Commenting Policies