Calculated Risk: Return on Security Investment

Jeff Nigriny wants to believe that patch management software is a good investment. but he can’t. until Nigriny, chief of security for aerospace and defense supply chain exchange network Exostar, can prove a positive return on his security investment, or ROSI, he will continue to manually patch systems. He will download the patches, perform regression testing, deploy them in a staging area, determine what machines need patches and then, finally, spit them out onto his network.

MORE ABOUT SECURITY AND FINANCIAL METRICS

• 2010: Infosec and the balanced scorecard
• 2008: Security and business 101: Financial basics
• 2008: Schneier: Security ROI, fact or fiction?
• 2008: Security metrics: Critical issues

“Patch management software seems like the perfect candidate to show an easy return,” says Nigriny. “Everyone kind of feels like it’s the right thing to do. But I haven’t procured a system. And I won’t—yet. Why? Because right now the ROSI for it isn’t working.”

He calls this particular scenario “the most difficult and abstract in terms of risk and return” that he’s worked on. It’s nothing like 24/7 monitoring, which he said was a cinch to bring to the brass, especially since after he proved an ROSI for monitoring, he also showed that he could cut costs another threefold by outsourcing it.

But with patching, he continues to build and then rebuild his ROSI models, looking for that elusive positive return, all the while fixing his systems the old-fashioned way.

Many of you might be snickering by now because you don’t share Nigriny’s idealism about the necessity of an ROSI to sell security to the CEO and CFO. In fact, it seems you are legion in your resistance.

It’s understandable, in a way. As CISO Tina LaCroix of insurance broker and consultancy Aon points out, “This elusive packaging of the ROI formula to validate our existence is one that may take us down an endless path,” a path that probably looks to many CSOs like the one Nigriny’s put himself on now with patch management.

But, in fact, it’s not an endless path, and we’re here to suggest not only that you can use ROSI to sell security internally but that you must. As good a reason as any for the mandate is this: Economist Frank Bernhard’s research shows about six cents of every revenue dollar is at risk due to a lack of information security, whereas many companies spend barely a dime of their IT dollar on security.

“I’m not sure why IT tends to disregard these tools; it’s a bit frustrating to keep hearing you can’t do it accurately,” says Bob Jacobson, founder and president of International Security Technology (IST), which handles physical and logical security risk assessment. “It’s not true. The tools are there. Nuclear uses them. Pharma uses them. The whole world has used ROI in security for a long time. [CSOs] have an opportunity to make a major contribution in their organization, if they have the willingness to learn this.”

None of which is to say ROSI isn’t hard work for a security executive; it is. But it’s not hard like calculusplenty of researchers and economists have taken care of sigmas and mus and other esoteric economic math already. It’s hard like running a marathonROSI requires legwork, and lots of it.

We’ll set you on the path to succeed in building and using ROSI as a tool to sell security, with a simple three-step primer. Trust us, your CEO will think it’s worth it.Step 1: Rethink your assumptionsExostar’s Nigriny is clearly not in the majority when it comes to security professionals and ROSI. The defeatist shrugs that accompany conversations about ROSI have become conventional wisdom. “Most execs want hard numbers to make financial decisions, and we live in a world where you can’t always have that,” says Rich Mogull, research director at Gartner G2 Cross-Industry Research. “I mean, what’s the ROI of a fire extinguisher?”

According to one study the American Society of Safety Engineers (ASSE) cites, the ROI of fire extinguishers is in fact about a $3 return for every $1 invested if you take fire extinguishers as part of a larger corporate health and safety initiativewhich you should, since fire extinguishers (like IT security) rarely show up as a discrete security purchase. (For the sake of our argument, ignore that Mogull’s example is hamstrung by the fact that, often, regulation mandates fire extinguishers.)

The point here is ROSI can be calculated and is being calculated. To do so with information security, though, there needs to be a deliberate effort to rethink some of the industry’s assumptions and cultural biases. Specifically, there are two biases that need to be eliminated:

Precision is not the goal. One of the reasons that ROSI might feel like an endless path comes from the fact that there has been a natural tendency in the tech sector toward approaching problems with the precision a software engineer would expect. The “hard numbers” Mogull assumes are required.

“This is a classic problem that technologists have,” says Kevin Soo Hoo, a researcher at security consultancy @Stake doing ROSI studies, and who at Stanford University wrote his thesis, dense with economic theory, on the subject. “They don’t understand that you can make rough guesses to work out a problem. We dive into an ROSI study, and the engineers are focused on the minutiae and want to argue for days whether some variable should be .6 or .55. It doesn’t matter,” Soo Hoo says emphatically, as if he’s been through this more than a few times. “Choose one!”

With ROSI, like all risk assessment, the goal instead needs to be accuracy, which isn’t at all the same thing as precision. Notice that the ASSE study suggested about $3 for every $1. There was no attempt here to delineate the exact return, because that’s not the point. The point is to provide a set of guiding principles from which you, your CEO and CFO can make good decisions about what’s acceptable. In other words, the CEO doesn’t (or shouldn’t) care if a return is precisely $3.13 for every $1 spent or $2.97. He cares that it’s accurate to suggest about a 3-to-1 return, and not a 1-to-1 return or, worse, a 1-to-3 return.

The dogmatic I.T. mind-set must be eliminated. It’s obvious why IT tends to approach problems with binary thinking. It is, after all, the language of the trade. But an on-off, “either we’ve been hacked or we haven’t” view of the problem will make ROSI an impossible task. (Some believe it helps to eliminate binary terms from their discussions so that security becomes risk management and threats aren’t eliminated, they’re mitigated and so forth.)

Back to the fire extinguishers. A binary thinker might suggest that, since there was no fire last year, there was no ROSI. If that is the attitude at your company, it’s time to initiate some awareness and education because that’s not how risk mitigation works. Think of it this way: If you wear your seat belt but don’t get in a car accident, does that mean you ought not invest in a seat belt because there was no return?

No. You did get a return, because return is not measured in a dogmatic world of what did or did not occur, but in the stochastic world of what might occur and how likely it is to occur. That is the game of risk; prepare for something to happen by investing in ways to stop it from happening.

“You can’t get from the cost of security incidents directly to a return on investment,” says Thomas Koulopoulos, president, CEO and founder of Delphi Group, an information technology research and consulting company. “You need to focus on the intermediate step. The probability.”Step 2: Do the legworkHere’s just a portion of the effort Nigriny put into his patch management ROSI: “I am throwing into it how many patches per year I apply, based on three years of data. I sit down with the network team and talk about the types of patches, their criticality level. I look at how long it takes to vet the patch. How many rollouts result in a rollback because of problems with the patch. Then I look at how many patches I should have installed, based on all the patches on all the mailing lists I subscribe to. I dedicate a day to that, but I could take weeks. Eventually, I come up with total time I was at X-percentage risk level before the patches were installed. Here’s the average cost of an incident to us; that’s my baseline number. You absolutely have to have that. There are industry baselines for this you can find. You can talk to peers at other companies about their baselines and massage them for your situation.”

You get the idea. ROSI is labor-intensive. In his partial history of the patch management ROSI above, though, Nigriny demonstrates much of what you need to do to prepare to use ROSI. Here it is:

Find and use data that’s out there. The most common misconception CSOs have about ROSI is that there isn’t any data available to even start an ROSI study. There’s a ton of it, and the body of usable statistics is growing. Some is free for the taking, other data you might have to pay for, but the actuarial figures do exist. (CSOs who come from a physical security world probably know this, as they’ve dealt with risk of theft, natural disasters and so forth for a long time and have sought out the data on the probability of such events.)

CERT and Riptech, for example, have combed over data to discover some incredibly useful facts. They measured attacks per company, which right now come in at a rate of 2,112 attacks over two years. What’s more, at current growth, that number will grow to 8,403 attacks per company over two years. That’s a fourfold increasewhich strengthens the ROI argument. Mitigation now will protect against a growing threat. In addition, CERT built some complicated math that shows security spending is a diminishing-return game; that is, as you spend more, the probability of attack goes down but at an ever-slowing rate. By crossing this data with what are called indifference curves (too complicated to get into here), you can actually determine a kind of sweet spot of security spending for your organization.

Consultancy @Stake has published well-known numbers that prove that the earlier you build security into applications, the higher the return. The company’s researchers now believe they probably lowballed their 21 percent ROI for incorporating security from the start.

You need to cull as much of this kind of data as possible and keep it in your toolbox because the more you set out to show returns on security, the more you’ll be coming back to these kinds of figures.

Canvass to get what’s not out there. If the first piece of advice is “go to the library,” then this is “play detective.” You must develop certain numbers, like the cost of incidents to your organization and the probability that a given incident will occur. While these numbers can be based on research, to hone them for your situation requires canvassing of the relevant playersincluding business managers within your company, peers at similar companies, economists, consultants and so on.

“My experience is that the business managers have clear ideas about loss, risk and what it will cost them and probably more experience than the security guys know,” says Jacobson of IST. “You have to go to Mr. Jones and ask him what it would cost him to be down, what is his optimum recovery time. He will have better answers than you think, especially as he thinks about it more.”

Know thyself. With all of this data in hand, you can start to build a threat profile. You’ll need to know the threats specific to your industry, the probabilities of certain types of attacks based on the kind of company you have or the kind of infrastructure you use. Crude but true example: Financial services companies face more attacks than manufacturing companies. Companies in the news endure spikes in attempted incidents. The Riptech statistics actually do some demographic breakdowns based on industry sector.

Calculate conservatively. We’re moving from how and where to get data to how you’re going to present it. When pulling together numbers for a ROSI study, always play it safe. Don’t assume costs or benefits you’re not sure of. If someone says the probability of an attack is between 10 percent and 20 percent, use 20 percent. If they say the cost of an attack is $50,000 to $100,000, take the bigger number.

And use “soft returns” as gravy. Soft returns are generally the hardest elements of a security investment to quantify. An improved brand image due to increased security is a soft return. Trying to add these to the equation is difficultsome skeptical CFOs might even dismiss your ROSI argument as “fudged” because of these variables. Therefore, soft returns are more effectively used as an added benefit on top of ROSI when selling executives.

Know your audience. And when selling the bosses, the CSO should learn what those executives are looking for in terms of return. “I can’t tell you how many times these things are rejected out of hand, because IT is selling something that the executives aren’t even looking to buy,” says Delphi’s Koulopoulos.

Know how the executives want the ROSI positionedcash savings, productivity gains, increase in securityand move forward that way. Many sources also report that making the ROSI case interactive for executivesallowing them to tweak variables and watch what happens to the ROSIis by far the single most effective selling tool you can use. “The key is not to be defensive about the data, as I think IT sometimes can be,” IST’s Jacobson says. “Don’t defend the model; explain it.”

Nigriny thinks there are other, underrated sales skills CSOs need to foster in themselves. A general familiarity with accounting is priceless, he says. Also, “You have to be good at public speaking and at PowerPoint engineering. If you’re speaking to the CFO, expect him to do some number crunching; have your numbers ready for him. The CEO? The executive summary is far more important. Talk to the CFO ahead of time; you’ll have his support, and the CEO won’t have to sit through the numbers discussion,” says Nigriny.

We weren’t kidding when we said this is laborious, intensive work. To Nigriny, ROSI is fractallike, in that the closer he examines his situation, the more intricate it becomes. “Every time I thought I had it covered, a raft of new variables came up. I’ve just got this swag of numbers here I have to deal with,” a nonplussed Nigriny says.

It’s up to the CSO to set the thresholds of what’s really needed for a particular scenario. You can make ROSI as simple or as complicated as you think is necessary, and an obvious tenet that emerges is that a simpler ROSI will be somewhat less accurate than a detailed ROSI, but the detailed version will require ever more legwork.

Step 3: Do the math

In the end, the math is simple. You subtract cost from benefits. A positive number is good: a return on investment. A negative number is bad: You’re spending more than you’re getting.

Of course, the math behind the variables and coefficients that go into the costs and benefits is massively complex. Fortunately, if you’ve got raw data from your legwork, someone else has done or will do the difficult computations for you. Still, there are some basic risk computations you should know. Here they are:

Annual Loss Expectancy. ALE is the foundation of risk assessment. It is what it sounds like: how much money you expect to lose per year due to some sort of security incident. Note that this is different than the raw cost of an incident (which, remember, you should always keep as a baseline). It’s actually the raw cost times the probability of an event in the next year. So the ALE of a security breach that costs $1 million and has a 40 percent chance of happening is:

Incident cost X Probability of incident = ALE

$1,000,000 X 0.4 = $400,000

Modified ALE. mALE is the same equation, but with the probability affected by mitigation measures you take. Imagine the above scenario were a virus attack. You introduce antivirus software that cuts in half the probability of a successful attack, to 20 percent. Or, you start an awareness program that reduces probability 5 percent. (These are arbitrary, but if you’ve done the legwork from Step 2, you’ll have real numbers to plug in here.) Then:

Probability X Mitigation A = Modified probability

Probability X Mitigation B = Modified probability

A: 0.4 X 0.5 = 0.2

B: 0.4 X 0.95 = 0.38

You must consider each mitigation separately. Once you’ve gone through the process for several types of mitigation, you can pick which ones you feel are most important or provide the best return. (Of course, some mitigation measures will have overlapping effects. We’re not putting that into this math.)

At any rate, adding mitigation measures produces modified ALEs:

Incident cost X Modified probability = mALE

A: $1,000,000 X 0.2 = $200,000

B: $1,000,000 X 0.38 = $380,000

So, in each case you’ve reduced your ALE.

ALE – mALE = Savings

A: $400,000 – $200,000 = $200,000

B: $400,000 – $380,000 = $20,000

This is the step at which executives will want to interact with the model, seeing how different measures that they take affect their mALE.

Now, to get a basic return, you simply subtract the cost to implement each mitigation measure from your savings on your mALE by implementing the mitigation. Let’s say mitigation A, antivirus software, costs $120,000. And mitigation B, an awareness program, costs $8,000. Then:

Savings – Mitigation cost = ROSI

A: $200,000 – $120,000 = $80,000

B: $20,000 – $8,000 = $12,000

Both mitigation measures provide a ROSI (if the final number came out negative, then you’re spending more than you’re getting back). Awareness actually has a higher return; or put another way, you get the most bang for the buck. (Your savings are 2.5 times what you spend, whereas in the antivirus case, they are 1.7 times what you spend.)

This is a simple model. No doubt CSOs, consultants and vendors with their own ideas will hue and cry that we’ve presented ROSI in this particular, facile way. But we’re only trying to provide a guiding primer. To attempt more in this space would be a fool’s errand. (For example, we didn’t even approach the concept of Net Present Value, which takes into account costs and benefits over time as if all the money were here now. Ask your CFO.)

Don’t take this as a final “how to” but rather as a starting point to develop your own ROSI. But don’t forget: The most important message is to do the homework. Collect as much data as possible so that there’s plenty to crunch.

ROSI is empirical, but in many ways it’s emotional, believe it or not. It is about coming up with numbers, but those numbers are only useful in the context of how executives feel about them. ROSI is risk economics that paints a picture of your organization’s attitude toward security. What level of risk is the enterprise comfortable with? How does the company prioritize its limited resources? Is technology or awareness more valuable as a tool? Suddenly you’re answering business questions based on the security numbers.

“The numbers right now show patch management automation doesn’t provide a positive return for this organization,” Nigriny says. “So why would I do it? It just doesn’t make sense.” Just by coincidence, it seems, ROSI has aligned Nigriny with the business.