• United States



by John Schroeter

Measuring the effectiveness of your security awareness program

Feb 12, 20146 mins
BudgetingData BreachIT Leadership

Organizations that maximize the efficacy of their security awareness programs stand to benefit not only from better incident prevention, but many other benefits as well, including improved reputation and customer loyalty, says John Schroeter

As Yogi Berra put it, “If you don’t know where you’re going, you’ll end up someplace else.” Do you know where you’re going with respect to your privacy and security awareness programs? How will you know when–or if–you get there?

[How to use Syrian Electronic Army attacks to improve security awareness]

“But wait just a minute,” you object. “Everyone knows that security is a process, not a destination. Is there really any such thing as arriving?” Well, of course there is. Just because a process is dynamic doesn’t mean it’s left without any measurable aspects. Besides, if any process is to be improved, it must also be measured.

There are many benefits an organization will enjoy when it makes those improvements, not the least of which is the budget justification for creating a security awareness program that help will boost security effectiveness overall. Martin Sadler, Director of Security at HP Labs, summed them up thusly: “Organizations that have achieved a high level of security effectiveness are better able to identify major data breaches, secure confidential information, limit physical access to data storage devices, and achieve compliance with legal and self-regulatory frameworks. They are also in a better position to attract and retain high-quality security personnel and enforce corporate policies.”

Those benefits have ripple effects throughout the organization—benefits that span protecting the company reputation to increasing customer trust and loyalty. And those translate directly to the bottom line.

Granted, measuring security effectiveness is not as straightforward as measuring a manufacturing process. There are many variables that are simply outside of one’s direct control. In fact, a recent ISACA report conceded, “…security is contextual and not an isolated discipline; it depends on the organization and its operations. Furthermore, effective security must take into account the dynamically changing risk environment within which most organizations are expected to survive and thrive.” All the more reason that improvements be addressed wherever possible!

In any case, this variability may explain the disparity of results Dr. Kenneth Knapp discovered when he investigated the effectiveness of security programs. He found that while the majority of infosec professionals surveyed believed they were able to secure their information effectively, only 22 percent of them believed so with a high degree of confidence.

[Punish careless employees to reduce security breaches, vendor says]

Moreover, the survey showed that more than a third did not believe that their organization effectively secures its data. And this is likely understated. Sounds like room for improvement.

When asked about this, Dr. Larry Ponemon of the Ponemon Institute admits that while security effectiveness can be an elusive object to measure, there are highly effective ways of determining it, short of recording incidents of catastrophic failure. So how do we go about making improvements? What, exactly, is it we can measure to determine whether the security awareness program is as effective as it ought to be? In answering these questions, Ponemon begins with identifying the key dimensions of information security effectiveness, which he describes as:

  • Uptime: The ability to withstand cyber attacks and avoid costly business disruption.
  • Compliance: The ability to achieve compliance with all applicable regulations and laws.
  • Threat containment: The ability to prevent or quickly detect external security threats such as cybercrime, social engineering or malicious attacks.
  • Cost efficiency: The ability to manage investments in information security and data protection in a competent (non-wasteful) manner.
  • Data breach prevention: The ability to prevent or quickly detect internal security threats such as the negligent or incompetent insider.
  • Policy enforcement: The ability to monitor and strictly enforce compliance with internal policies, procedures and other security requirements.

These are the metrics Ponemon applied when he developed his breakthrough Security Effectiveness Score, which, in its most compact version, evaluates 24 attributes (extrapolated from the six key dimensions described above) that consistently correlate with strong security postures. In short, the higher the score, the stronger the organization’s security posture, the greater its ability to avoid a breach, and the lower the cost to mitigate a breach. In other words, an objective standard of measure for security effectiveness.

One of the most significant insights that resulted from the application of the tool is that of the 24 parameters considered, 75 percent of them are directly related to security-aware behaviors, not just information technology. And when specific employee behaviors are addressed in a meaningful way to bring about a security-aware culture, the incidence and cost of non-compliance plummets.

[How to create security awareness with incentives]

More importantly, when correlating the scores to actual breach incidents, Ponemon’s data (gleaned from the more than 7,000 security effectiveness score surveys he’s collected) also demonstrates that when organizations spend a dollar on information security–and particularly on security awareness training–they get far more than a dollar’s value in return. In other words, an ROI. Another reason to see how your organization measures up!

Lastly, ISACA points out in “Security Awareness: Best Practices to Secure Your Enterprise” that measurement not only reveals whether the awareness program is effective, but it can also help identify any knowledge gaps and ensure the improvement of the program overall. Surveys, interviews, pop quizzes, exams, and audits are a few of the more common assessment tools that can be used to measure progress.

A case in point is Western Union’s approach to measuring the results of its security awareness program. Western Union’s Kim Hickman explains, “Of course you always wonder if you’re making an impact, if your efforts are paying off. So to gauge and quantify that we started conducting 20-question quizzes, sent to a different sampling of the employee population every month. We trend the scores over time to see if, as an organization, we’re getting better. And we have seen improvement since we launched our new security course, with quiz scores now averaging 89%. It is definitely raising awareness and changing behavior.”

Furthermore, Hickman also observes that the quizzes have the additional effect of reinforcing the security awareness information presented in their course. “We get a double benefit there,” she says.

The bottom line? The very act of measuring actually also helps bring about the desired result!

John Schroeter is a security awareness programs strategist.