Oft-cited cybercrime cost estimates hosed down

President Obama said it in a major speech on cybersecurity. U.S. Senators said it while promoting their Cyber Security Act of 2012. Gen. Keith Alexander, director of the National Security Agency and head of the U.S. Cyber Command said it while warning of “the greatest transfer of wealth in history,” through the theft of intellectual property.

But because they cited it — the estimated cost of cybercrime — that doesn’t make it true, a ProPublica report says.

There is general agreement that the worldwide cost of cybercrime is in the hundreds of billions. But just how many hundreds of billions is a matter of debate, following ProPublica’s report that questioned the most widely quoted estimates by two major security vendors.

McAfee has estimated the annual cost of cybercrime worldwide at $1 trillion; Symantec has estimated the annual cost of intellectual property theft in the U.S. at $250 billion.

The report says they are exaggerations — perhaps vast — noting that the $1 trillion figure was not even in the actual McAfee report, but in the press releases about it.

They are not the only ones calling such estimates into question. Computer scientists Dinei Florencio and Cormac Herley, of Microsoft Research, authors of a recent paper, titled “Sex, Lies and Cyber-crime Surveys,”, who wrote, “Our assessment of the quality of cyber-crime surveys is harsh: they are so compromised and biased that no faith whatever can be placed in their findings.”

The ProPublica report saus the McAfee estimate is disputed by some of those who analyzed data for the 2009 report, which was based on information gathered from a survey of 1,000 IT professionals.

Eugene Spafford, one of three independent researchers from Purdue University, told them: “I was really kind of appalled when the number came out in news reports, the trillion dollars, because that was just way, way large.”

Another researcher, Ross Anderson, a security engineering professor at University of Cambridge, told ProPublica he “would have objected at the time had I known about [the $1 trillion estimate.] The intellectual quality of this is below abysmal.”

“[The Symantec estimate] was indeed mentioned in a Symantec report, but it is not a Symantec number and its source remains a mystery,” the report said.

Sal Viveros, a McAfee public relations official who oversaw the 2009 report, had not responded to a request for comment by the deadline for this story. But he wrote in an email to ProPublica: “We work with think tanks and universities to make sure our reports are non-biased and as accurate as possible.”

Other security experts and analysts tend to agree with ProPublica, saying not only that the estimates are inflated but that any estimate from a security vendor should be treated with some skepticism, because there is a built-in conflict of interest — the worse the security risks and costs are, the better it is for their business.

In addition, industry reports are not subject to the kind of peer review that is done for academic and professional journals.

But experts are also willing to cut the companies some slack, for a couple of reasons. First, it is very difficult to estimate such things. Sometimes, companies don’t even know they have been attacked. Many times, when they find out, they don’t want to talk about it, lest they damage their brand. And sometimes it is difficult to know how much actual damage has occurred.

“I don’t beat them up for it,” said Jason Healey, of the Atlantic Council and a former White House and Goldman Sachs security official. “Experts have long had trouble agreeing on estimates that are within even two orders of magnitude of each other.”

Healey said the damage estimates of the first large-scale cyber incident, the Morris worm of 1988 “ranged from $200 to more than $53,000 per installation, while the most widely cited estimate of the total damage ranged from $100,000 to $10 million: two full orders of magnitude. And that was 24 years ago.”

Gary McGraw, CTO of Cigital, said he suspects McAfee “followed protocol [in its report] up to the end, where they did some crazy math — I think control got turned over to the marketing guys.”

But he admits, “I’ve cited that [$1 trillion] number in my own work. I was writing a piece about cyberwar for a think tank. I was trying to make a point about cybercrime being worse than cyberwar — which the risks of cyberwar were exaggerated, and cybercrime was worse. How’s that for irony?”

There are other reasons that estimates are difficult. In a recent paper called “Measuring the Cost of Cyber Crime,” done for the UK Ministry of Defense, the authors listed a chart that suggested the annual cost of worldwide cybercrime was about $225 billion — less than 25% of the McAfee estimate.

But the authors included a host of caveats, including: “There are over 100 different sources of data on cybercrime, yet the available statistics are still insufficient and fragmented; they suffer from under- and over-reporting, depending on who collected them, and the errors may be both intentional (e.g., vendors and security agencies playing up threats) and unintentional (e.g., response e