Why metrics are crucial to proving cybersecurity programs’ value

Methodologies to measure the effectiveness of cybersecurity efforts exist. Tying them to the real world is the trick.

dashboard / report / metrics / results / analysis  / management
Peshkov / Getty Images

As solutions to managing cybersecurity threats increase, surprisingly few metrics are available on how well these methods work to secure organizational assets. The National Institute of Standards and Technology (NIST) has pioneered information security performance measurement models that can produce metrics. (Note: NIST’s work in this area is now being updated.)

Aside from government agencies’ requirements to produce information security performance measures, the measurement models NIST recommends can also be used for internal overall IT improvement efforts. Either way, NIST recommends considering four factors while developing and implementing an information security measurement program:

  • Quantifiable measures
  • Readily available data that support the measures
  • Repeatable information security processes
  • Utility for tracking performance and directing resources

As is true of many NIST cybersecurity efforts, its information security performance measurements lack real-world implementation guidance that could assist technologists in measuring security performance, leaving the industry struggling for pragmatic advice. Speaking at Shmoocon last week, Robert Weiss, head of information security at OpenVPN, tried to fill the void by providing security professionals with practical ideas on starting their information security metrics programs.

The most critical measurement is risk

“No metrics presentation has ever been funny, and this one is no exception,” Weiss said. All jokes aside, he stressed that metrics are crucial to effective cybersecurity programs despite how rarely organizations do a good job or make any effort to rely on them. “If our job as information security professionals is to reduce information security risk, at the end of the day if we can't demonstrate that we're accomplishing this objective, resources will and should go elsewhere.”

The most important thing to measure is risk. “Our programs are designed to reduce risk,” Weiss said. “The relationship of the program’s cost to the amount of risk reduction is the business value being created.” But measuring risk reduction isn’t the only goal of a security metrics program. “We may often do other things like program performance or create situational awareness,” he added.

“In a perfect world, you would have systems and processes for tracking performance, situational awareness, and risk. You track metrics that matter. You do not rely on surveys. You pull empirical data from your systems and reason about your uncertainty and margin error.”

Ideally, “you can express risk in the probability that the annual loss expectancy for a series of risks falls within a particular range. You immerse yourself in the language of probability,” Weiss added. “Very few organizations can do this. This actually represents a huge opportunity for both practitioners like yourselves and your businesses.”

Two basic security metrics methodologies

Weiss emphasized two primary methodologies to help security professionals establish metrics programs. The first is “just measure everything.” Collecting everything “sends the message that you plan to build the culture of measurement and make decisions on facts and analysis.”

There is a point of diminishing returns in this methodology. “If you have no data, any new data will greatly expand your knowledge and reduce uncertainty,” Weiss said. However, “there's an interesting corollary. If you have a lot of data adding more isn't going to be very valuable.” You want to spend just enough to collect data that will help make decisions, but not more, he added.

If the data doesn’t exist, you can estimate it using secondary sources. “Most of the time, you don't need a lot of data to make management decisions. You can test a sample of servers. You can use secondary sources like the Verizon breach report or others to get information about types, incidents, and losses,” Weiss said.

The second methodology calls for collecting data and then applying analytical techniques that help describe the information’s nature. Weiss relied on the classification systems of psychologist Stanley Smith Stevens who created the classic measurement scales of nominal, ordinal, interval, and ratio in spelling out the merits of this approach.

“It is very common in information security to see a system or probability impact plotted in some form of matrix because probability times impact equals risk,” Weiss said. But, the dangers of the analytical approach come into play, for example, “when you try to compare two ordinal scales [e.g., small, bigger, biggest] to each other. It is impossible to relate one arbitrary step of probability to one arbitrary step of impact. Those things cannot and should not be related without additional information. It’s like multiplying by color.”

Don’t count only adversary incidents

Metrics programs should follow strategic goals and avoid certain traps that undermine organizational security, Lesley Carhart, director of incident response for North America at Dragos, tells CSO.  One of those traps is when security metrics are based on adversary activity.

“You can’t predict reliably in the cybersecurity space when somebody is going to attack or how often they’re going to attack,” Carhart says. “And if they base their success on the number of incident responses they do or the number of tickets that they handle based on adversary activity, what happens if an adversary doesn’t attack that month? Or if they attack more in one month than another?”

“It's non-sensical to base your measures of success on when a criminal does something that’s completely unpredictable,” Carhart says. “You have to really understand what you’re measuring. You don’t just do KPIs [key performance indicators] for KPIs’ sake. It’s incredibly problematic. That’s why we get unhealthy things like these phishing test scenario programs.” Instead of, for example, clickbait rates, a better “measure is how often people want to report things. Because just one campaign report could tip you off and let you do your cybersecurity much faster.”

The phishing example highlights why “you don’t want to base any of your metrics on whether a bad person attacks or not,” Carhart says. “Make sure none of your measures are based on that. And think critically about what you are actually trying to accomplish, your organization’s goals, and base your metrics around that.”

Weiss agrees but tells CSO he wants all the numbers to start making decisions as a CISO about which ones are the most important. The important thing is to “make a commitment to data analytics,” Weiss stresses. “And you don’t have to do everything perfectly.”

Copyright © 2022 IDG Communications, Inc.

22 cybersecurity myths organizations need to stop believing in 2022