A Few Good Information Security Metrics

Andrew Jaquith says information security metrics don't have to rely on heavy-duty math to be effective, but they also don't have to be dumbed down to red, yellow, green. Here are five smart measurements--and effective ways to present them.

July 01, 2005 — CSO — Information security metrics have a bad rep. Mention metrics to a CISO and immediately his thoughts may well turn to sigmas, standard deviations and, probably, probability. To many, metrics equals statistics.

There's no denying that proven economic principles can, and should, be applied to information security investments. At the same time, a bumper crop of valuable metrics exist that don't require classes on Nobel Prize-winning theories or a working knowledge of the Greek alphabet. You've actually already sowed the seeds of these less dense but equally valuable metrics. They're sitting in your log files, on your network, in the brains of your business unit managers, just waiting to be harvested. You won't need computational prowess to exploit this crop's value, just some legwork and—this is key—the most effective presentation tools.

Here we discuss five such metrics, along with some ways to present them visually, as imagined by Andrew Jaquith. Jaquith is a cofounder of the consultancy @stake (which was bought in 2004 by Symantec) and a protege of infosecurity guru Dan Geer. At @stake he invented a popular analytic methodology that is used to evaluate a client's risk in its application portfolio. He's since left Symantec and joined The Yankee Group. More recently he started Securitymetrics.org, a website open to all security professionals for sharing, contributing and advancing the use of metrics in information security. He has also writing a book, Security Metrics: Replacing Fear, Uncertainty, and Doubt, due out later this year.

Jaquith has sharp, sometimes contrarian opinions on what makes a good metric and what makes for good presentation of metrics. For example, he thinks annual loss expectancy (ALE), a tool used to measure potential losses against probability of losses occurring over time, is useless, because in infosecurity, the L and the E in ALE are wild guesses. Quoting Geer, he says, "The numbers are too poor even to lie with."

He also thinks CISOs are too apt to dumb down visual representations of metrics for their executive counterparts, mistaking simplicity for clarity. He holds a particular grudge against the overuse of the "red, yellow, green" representation of metrics to signify high, medium and low numbers. "A CEO's favorite visualization of metrics is a stock chart, a 1-inch square that contains a month's worth of opening and closing prices, a trend line and several other indicators. Maybe 50 or more data points right there. Don't tell me they can't handle complex data. They can, as long as it's presented well."