The great IT risk measurement debate, part 2

IT risk—can it be measured, modeled, mitigated? Part two of Alex Hutton and Douglas Hubbard's discussion covers likelihood statements, the placebo effect on risk perception, and much more.

By Bill Brenner

March 02, 2011 — CSO —

For the beginning of this discussion, see part one. We left off with Alex Hutton pointing out the problem with creating rigid risk management requirements when the measurement of risk is flawed to begin with.

Alex Hutton: We need something that's much more flexible, where we have dozens of models for dozens of uses in the management of risk that may be informative independent of [identifying] likelihood and impact. One of the greatest things that happened to me moving from Risk Management Insight over to Verizon was that Verizon had a completely different, epidemiologically based view of risk—"Let's go to the source of things that were incidents, and let's do a study."

Dr. [Peter] Tippett [vice president of industry solutions and security practices at Verizon Business] is a medical doctor and a Ph.D., with years of epidemiological experience, so his approach was, "Let's start gathering data. Let's build the framework by what data we need to extract out of that, what's meaningful and so forth." And this just blew my mind—a completely different view of risk and what creates it. And so I would rather, as I said, that people focus on gathering data and then looking for interesting correlations and using those to be actionable, rather than worrying about whether or not their likelihood statements are really reflective of reality.

At some point I think we can get to that, and probably much sooner rather than later, but right now, no standard is helping anyone do anything that would stand up to any scrutiny in any of your books. [There's no standard] where, in my eyes or in the eyes of any professional, really, you would say, "Aha! You have a decent model."

Douglas Hubbard: There's a couple of ways to compare these sorts of things [such as security incidents]. One is, if you look at a whole system, can we take a whole bunch of examples of different organizations using some methodology, and can we have a large enough number of trials over a longitudinal study that would show that you're better off with one method than with another, by looking at the shareholder value of the firms and things like that?

Read much more CSOonline coverage of critical issues in security metrics

Since there hasn't been anything like that for infosec yet that I'm aware of, the other approach is component testing: We can at least be sure whether or not the gears of the clock work.

Hutton: That's where I was going to go. That's one of the reasons why I think everybody should read your book rather than worry about getting an industry certification if they're really interested [in risk management].

Hubbard: Look at likelihood statements. There is such a large number of likelihood statements for an individual. I mean, if you start capturing all of them over the course of a year, in a year or two an organization will have a very large number of likelihood statements for a large number of analysts. And if you looked at all the likelihood statements and someone was saying that "these things are 10 percent likely per year," well, about 10 percent of those events should have happened in the course of the last 12 months so.

• Print