Cybercrime surveys a crock? Yes and no

Microsoft researchers Dinei Florencio and Cormac Herley have the audacity to question the accuracy of most cybercrime surveys. But are their conclusions right?

Yes.

And no.

Let’s start with their main thesis, as outlined in the story written by my colleague John E Dunn:

Cybercrime surveys are so riddled with statistical distortions the answers they glean are probably about as reliable as asking the average American Don Juan how many sexual partners he’s slept with, a Microsoft research paper has calculated.

In Sex, Lies and Cybercrime Surveys authors Dinei Florencio and Cormac Herley use the analogy of common errors found in sex surveys to illustrate a similar problem in many cybercrime studies.

The problem relates to the extrapolation from “heavytail” statistical means. When estimating an unknown quantity (cybercrime losses for instance) from subjective responses, small numbers of unusual answers can hugely distort the result if this result is taken to be representative of a whole population’s experience.

Sign up today.

Get your morning news fix with the daily Salted Hash e-newsletter!

In sex surveys, most men and women tell the truth when asked how many partners they have slept with, but a small number (especially men) are inclined to exaggerate to such an extent that it can badly skew mean statistics in ways that are difficult to detect.

The equivalent in cybercrime surveys would be the very high losses experienced by a small number of individuals being generalised across larger groups which have overwhelmingly suffered only mild distress, offering a distorted picture of the experience of most victims.

It is an obvious point that subjective surveys have this sort of risk built into them, but that hasn’t stopped the security industry wielding surveys in a tool of knowledge and understanding when they are nothing of the sort.

As the authors point out, the least reliable cybercrime numbers tend to be the “eye-popping” estimates of financial losses sometimes attributed to different types of cybercrime.

“It does not appear generally understood that the estimates we have of cybercrime losses also have these ingredients of catastrophic error, and the measures to safeguard against such bias have been universally ignored,” say the authors.

“Our assessment of the quality of cyber-crime surveys is harsh: they are so compromised and biased that no faith whatever can be placed in their findings. We are not alone in this judgement.”

To read the full 11-page report, open this .pdf document.

As someone who has written countless articles about security surveys, I first reacted to this report with a loud “Duh.” But there is something more worth saying here.

That surveys are often biased toward the vendor commissioning them is nothing new. It’s also not new that most surveys have numbers that are too open to interpretation to take too seriously.

If one survey of 1,000 security practitioners is done and half of them say something is good or bad, it’s not going to be a realistic picture of an industry as large as this one.

Looking back, there are some surveys I’ve had to write about that were particularly off the wall. One was a Yankee Group study in 2004 that predicted that “by 2010 most companies will outsource 90 percent of their security functions.” The most commonly outsourced items would technology for firewall management, network monitoring and vulnerability management.

At the time, I thought the 90 percent figure was way over the top. I don’t have an updated percentage today, though I’m working to get some fresh research on the subject.

I know from the surveys we’ve done with PriceWaterhouseCoopers that a lot of conflicting information has emerged on the outsourcing picture in a relatively short span of time.

When writing about the 2009 survey Global Security Survey with PWC, the picture was that outsourcing was in decline and companies were doing more in house:

A few years ago, technology analysts were predicting unlimited growth for managed security service providers (MSSPs). Many companies then viewed security as a foreign concept, but laws such as Sarbanes-Oxley, the Health Insurance Portability and Accountability Act (HIPAA) and the Gramm-Leach-Bliley Act (affecting financial services) were forcing them to address intrusion defense, patch management, encryption and log management. Convinced they couldn’t do it on their own, companies chose outsourcers to do it for them. Gartner estimated the MSSP market in North America alone would reach $900 million in 2004 and that it would grow another 18 percent by 2008.

Then came the economic tsunami, which appears to have cast a shadow over outsourcing plans even though security budgets are holding steady. Although 31 percent of respondents this year are relying on outsiders to help them manage day-to-day security functions, only 18 percent said they plan to make security outsourcing a priority in the next 12 months.

When it comes to specific functions, the shift has already begun. Last year, 30 percent of respondents said they were outsourcing management of application firewalls, compared to 16 percent today. Respondents cited similar reductions in outsourcing of network and end-user firewalls. Companies have also cut back on outsourcing encryption management and patch management.

At the same time, more companies are spending money on these and other security functions. Sixty-nine percent said they’re budgeting for application firewalls, up slightly compared to the past two years. Meanwhile, more than half of respondents said they are investing in encryption for laptops and other computing devices.

That, by the way, was based on a survey of 7,200 business and technology executives worldwide.

A year later, we did the survey again, with 12,847 business and technology executives from around the world responding. This time, the outsourcing trend seemed to be reversing itself again.

IT and business leaders acknowledge they don’t have the staff or expertise to secure their data internally — at least not without help from outside experts. If you work for a managed security service provider (MSSP), that’s good news.

That’s one of the takeaways from the Eighth Annual Global Information Security Survey CSO conducted along with sister publication CIO and PriceWaterhouseCoopers. Some 12,847 business and technology executives from around the world took the survey.

More than half (52 percent) of survey respondents said that outsourcers, also known as managed security service providers (MSSPs), are important or very important to accomplishing their security objectives. Another 19 percent said outsourcers play some role. Meanwhile, more than 30 percent cited outsourcing of some or all security functions, such as e-mail filtering and management of application firewalls, as a top priority in the next 12 months, up from 18 percent a year ago.

At that point, I decided to stop believing any numbers that came our way on the subject of security outsourcing.

But it’s a reflection of security surveys in general: It’s too easy to bend and shape the numbers to suit someone’s point. Which means it would be foolish to take any such survey as Gospel.

But it would be equally foolish to ignore surveys altogether.

I’ve often been asked why I continue to write about surveys even though I’m skeptical of them. The answer is easy:

You can never accurately capture the big picture with these things. But you can always pull smaller truths out of the mush, and writing about them can be helpful to those dealing with the specific challenges in question.

Let’s go back to the outsourcing issue.

While I don’t think we captured the big picture, we did capture the fact that different enterprises have different challenges.

Some are doing more in house because it’s more cost-effective for them do do so. Some find it’s too expensive to expand the in-house IT security staff. Each have points to make and the reader can take those points, decide where they fit in and act accordingly.

As for the charge that respondents often exaggerate or understate, as in the case of sex surveys and how many partners someone has slept with, that will always be the case and the reader should be prepared for that.

The big point here is that most surveys are flawed in one way or another. None will give you the perfect picture of reality.

But there will always be little nuggets of experiences to draw from that have value.

The key is to understand that going in.

–Bill Brenner

one-stop view of latest business threats. We created it for you! Bookmark it! Use it!

CSO’s Daily Dashboard gives you a