Home » Data Protection » Network Security

Research

Using Metrics to Diagnose Problems: A Case Study

Andrew Jaquith is a Yankee Group analyst and founder of discussion site Securitymetrics.org. The following excerpt is taken from his current book, Security Metrics: Replacing Fear, Uncertainty, and Doubt.

By Andrew Jaquith

Page 2

Diagnostic questions generally take the form of "The number of X is greater (or less) than Y" or "The percentage of X is greater (or less) than Y." For example, "There are no open wireless access points that can be accessed from the building's parking lot or surrounding areas" or "100% of the wireless access points on the corporate LAN require 128-bit WPA security." The diagnostic questions dictate our metrics. The primary benefit of the diagnostic method is that hypotheses are proven or disproven based on empirical evidence rather than intuition. Because each hypothesis supports the other, the cumulative weight of cold, hard facts builds a supporting case that cannot be disputed. A secondary benefit of the diagnostic method is that it forces the analysis team to focus only on measurements that directly support or disprove the overall hypothesis. Extraneous "fishing expeditions" about theoretical issues that cannot be measured automatically filter themselves out.

So far, the sample hypotheses and diagnostic questions I have given are rather simplistic. Why don't we return to our friend Barry's company for a real-world example?

Recall that Barry's original question was "Is my company's customer data secure from outside attack?" Our overall hypothesis held that, indeed, the company was highly vulnerable to attack from outsiders. To show that this statement was true (or untrue), we constructed subhypotheses that could be supported or disproven by asking specific questions whose answers could be measured precisely and empirically. The table above shows a subset of the diagnostics we employed to test the hypothesis. Note that these diagnostics do not exhaust the potential problem space. Time and budget impose natural limits on the diagnostics that can be employed.

To answer the diagnostic questions we posed, we devised a four-month program for Barry's company. We assessed their network perimeter defenses, internal networks, top ten most significant application systems, and related infrastructure. When we finished the engagement and prepared our final presentation for Barry, his team, and the company's management, the metrics we calculated played a key role in proving our hypothesis. The evidence was so compelling, in fact, that the initial engagement was extended into a much longer corrective program with a contract value of several million dollars.