Jeff Nigriny wants to believe that patch management software is a good investment. but he can't. until Nigriny, chief of security for aerospace and defense supply chain exchange network Exostar, can prove a positive return on his security investment, or ROSI, he will continue to manually patch systems. He will download the patches, perform regression testing, deploy them in a staging area, determine what machines need patches and then, finally, spit them out onto his network.
MORE ABOUT SECURITY AND FINANCIAL METRICS
- 2010: Infosec and the balanced scorecard
- 2008: Security and business 101: Financial basics
- 2008: Schneier: Security ROI, fact or fiction?
- 2008: Security metrics: Critical issues
"Patch management software seems like the perfect candidate to show an easy return," says Nigriny. "Everyone kind of feels like it's the right thing to do. But I haven't procured a system. And I won't—yet. Why? Because right now the ROSI for it isn't working."
He calls this particular scenario "the most difficult and abstract in terms of risk and return" that he's worked on. It's nothing like 24/7 monitoring, which he said was a cinch to bring to the brass, especially since after he proved an ROSI for monitoring, he also showed that he could cut costs another threefold by outsourcing it.
But with patching, he continues to build and then rebuild his ROSI models, looking for that elusive positive return, all the while fixing his systems the old-fashioned way.
Many of you might be snickering by now because you don't share Nigriny's idealism about the necessity of an ROSI to sell security to the CEO and CFO. In fact, it seems you are legion in your resistance.
It's understandable, in a way. As CISO Tina LaCroix of insurance broker and consultancy Aon points out, "This elusive packaging of the ROI formula to validate our existence is one that may take us down an endless path," a path that probably looks to many CSOs like the one Nigriny's put himself on now with patch management.
But, in fact, it's not an endless path, and we're here to suggest not only that you can use ROSI to sell security internally but that you must. As good a reason as any for the mandate is this: Economist Frank Bernhard's research shows about six cents of every revenue dollar is at risk due to a lack of information security, whereas many companies spend barely a dime of their IT dollar on security.
"I'm not sure why IT tends to disregard these tools; it's a bit frustrating to keep hearing you can't do it accurately," says Bob Jacobson, founder and president of International Security Technology (IST), which handles physical and logical security risk assessment. "It's not true. The tools are there. Nuclear uses them. Pharma uses them. The whole world has used ROI in security for a long time. [CSOs] have an opportunity to make a major contribution in their organization, if they have the willingness to learn this."
None of which is to say ROSI isn't hard work for a security executive; it is. But it's not hard like calculus
We'll set you on the path to succeed in building and using ROSI as a tool to sell security, with a simple three-step primer. Trust us, your CEO will think it's worth it.Step 1: Rethink your assumptionsExostar's Nigriny is clearly not in the majority when it comes to security professionals and ROSI. The defeatist shrugs that accompany conversations about ROSI have become conventional wisdom. "Most execs want hard numbers to make financial decisions, and we live in a world where you can't always have that," says Rich Mogull, research director at Gartner G2 Cross-Industry Research. "I mean, what's the ROI of a fire extinguisher?"
According to one study the American Society of Safety Engineers (ASSE) cites, the ROI of fire extinguishers is in fact about a $3 return for every $1 invested if you take fire extinguishers as part of a larger corporate health and safety initiative
The point here is ROSI can be calculated and is being calculated. To do so with information security, though, there needs to be a deliberate effort to rethink some of the industry's assumptions and cultural biases. Specifically, there are two biases that need to be eliminated:
Precision is not the goal. One of the reasons that ROSI might feel like an endless path comes from the fact that there has been a natural tendency in the tech sector toward approaching problems with the precision a software engineer would expect. The "hard numbers" Mogull assumes are required.
"This is a classic problem that technologists have," says Kevin Soo Hoo, a researcher at security consultancy @Stake doing ROSI studies, and who at Stanford University wrote his thesis, dense with economic theory, on the subject. "They don't understand that you can make rough guesses to work out a problem. We dive into an ROSI study, and the engineers are focused on the minutiae and want to argue for days whether some variable should be .6 or .55. It doesn't matter," Soo Hoo says emphatically, as if he's been through this more than a few times. "Choose one!"
With ROSI, like all risk assessment, the goal instead needs to be accuracy, which isn't at all the same thing as precision. Notice that the ASSE study suggested about $3 for every $1. There was no attempt here to delineate the exact return, because that's not the point. The point is to provide a set of guiding principles from which you, your CEO and CFO can make good decisions about what's acceptable. In other words, the CEO doesn't (or shouldn't) care if a return is precisely $3.13 for every $1 spent or $2.97. He cares that it's accurate to suggest about a 3-to-1 return, and not a 1-to-1 return or, worse, a 1-to-3 return.
The dogmatic I.T. mind-set must be eliminated. It's obvious why IT tends to approach problems with binary thinking. It is, after all, the language of the trade. But an on-off, "either we've been hacked or we haven't" view of the problem will make ROSI an impossible task. (Some believe it helps to eliminate binary terms from their discussions so that security becomes risk management and threats aren't eliminated, they're mitigated and so forth.)
Back to the fire extinguishers. A binary thinker might suggest that, since there was no fire last year, there was no ROSI. If that is the attitude at your company, it's time to initiate some awareness and education because that's not how risk mitigation works. Think of it this way: If you wear your seat belt but don't get in a car accident, does that mean you ought not invest in a seat belt because there was no return?
No. You did get a return, because return is not measured in a dogmatic world of what did or did not occur, but in the stochastic world of what might occur and how likely it is to occur. That is the game of risk; prepare for something to happen by investing in ways to stop it from happening.
"You can't get from the cost of security incidents directly to a return on investment," says Thomas Koulopoulos, president, CEO and founder of Delphi Group, an information technology research and consulting company. "You need to focus on the intermediate step. The probability."Step 2: Do the legworkHere's just a portion of the effort Nigriny put into his patch management ROSI: "I am throwing into it how many patches per year I apply, based on three years of data. I sit down with the network team and talk about the types of patches, their criticality level. I look at how long it takes to vet the patch. How many rollouts result in a rollback because of problems with the patch. Then I look at how many patches I should have installed, based on all the patches on all the mailing lists I subscribe to. I dedicate a day to that, but I could take weeks. Eventually, I come up with total time I was at X-percentage risk level before the patches were installed. Here's the average cost of an incident to us; that's my baseline number. You absolutely have to have that. There are industry baselines for this you can find. You can talk to peers at other companies about their baselines and massage them for your situation."
You get the idea. ROSI is labor-intensive. In his partial history of the patch management ROSI above, though, Nigriny demonstrates much of what you need to do to prepare to use ROSI. Here it is:
Find and use data that's out there. The most common misconception CSOs have about ROSI is that there isn't any data available to even start an ROSI study. There's a ton of it, and the body of usable statistics is growing. Some is free for the taking, other data you might have to pay for, but the actuarial figures do exist. (CSOs who come from a physical security world probably know this, as they've dealt with risk of theft, natural disasters and so forth for a long time and have sought out the data on the probability of such events.)
CERT and Riptech, for example, have combed over data to discover some incredibly useful facts. They measured attacks per company, which right now come in at a rate of 2,112 attacks over two years. What's more, at current growth, that number will grow to 8,403 attacks per company over two years. That's a fourfold increase
Consultancy @Stake has published well-known numbers that prove that the earlier you build security into applications, the higher the return. The company's researchers now believe they probably lowballed their 21 percent ROI for incorporating security from the start.
You need to cull as much of this kind of data as possible and keep it in your toolbox because the more you set out to show returns on security, the more you'll be coming back to these kinds of figures.
Canvass to get what's not out there. If the first piece of advice is "go to the library," then this is "play detective." You must develop certain numbers, like the cost of incidents to your organization and the probability that a given incident will occur. While these numbers can be based on research, to hone them for your situation requires canvassing of the relevant players
"My experience is that the business managers have clear ideas about loss, risk and what it will cost them and probably more experience than the security guys know," says Jacobson of IST. "You have to go to Mr. Jones and ask him what it would cost him to be down, what is his optimum recovery time. He will have better answers than you think, especially as he thinks about it more."
Know thyself. With all of this data in hand, you can start to build a threat profile. You'll need to know the threats specific to your industry, the probabilities of certain types of attacks based on the kind of company you have or the kind of infrastructure you use. Crude but true example: Financial services companies face more attacks than manufacturing companies. Companies in the news endure spikes in attempted incidents. The Riptech statistics actually do some demographic breakdowns based on industry sector.
Calculate conservatively. We're moving from how and where to get data to how you're going to present it. When pulling together numbers for a ROSI study, always play it safe. Don't assume costs or benefits you're not sure of. If someone says the probability of an attack is between 10 percent and 20 percent, use 20 percent. If they say the cost of an attack is $50,000 to $100,000, take the bigger number.
And use "soft returns" as gravy. Soft returns are generally the hardest elements of a security investment to quantify. An improved brand image due to increased security is a soft return. Trying to add these to the equation is difficult
Know your audience. And when selling the bosses, the CSO should learn what those executives are looking for in terms of return. "I can't tell you how many times these things are rejected out of hand, because IT is selling something that the executives aren't even looking to buy," says Delphi's Koulopoulos.