Using Metrics to Diagnose Problems: A Case Study

Andrew Jaquith is a Yankee Group analyst and founder of discussion site Securitymetrics.org. The following excerpt is taken from his current book, Security Metrics: Replacing Fear, Uncertainty, and Doubt.

By Andrew Jaquith

May 17, 2007 — CSO —

A few years ago my former employer was called in by the CTO of a large, well-known maker of high-end consumer electronics. This company, which prides itself on its progressive approach to IT management, operates a large, reasonably up-to-date network and a full suite of enterprise applications. The CTO, Barry Eiger (a pseudonym), an extremely smart man, is fully conversant in the prevailing technology trends of the day. In manner and in practice, he tends to be a conservative technology deployer. Unimpressed with fads and trends, he prefers to hydrofoil above the choppy technological seas with a slightly bemused sense of detachment. Facts, rather than the ebbs and flows of technology, weigh heavily in his decision-making. In our initial conversations, he displayed an acute awareness of industry IT spending benchmarks. We discovered later that he had spent significant sums of money over the years on advisory services from Gartner Group, Meta Group, and others.

If he is so well informed, why did he call us in, I wondered? Barry's problem was simple. His firm had historically been an engineering-driven company with limited need for Internet applications. More recently, his senior management team had asked him to deploy a series of transactional financial systems that would offer customers order management, loan financing, and customer support services. These public-facing systems, in turn, connected back to several internal manufacturing applications as well as to the usual suspects—PeopleSoft, SAP, Siebel, and Oracle. A prudent man, Barry wanted to make sure his perimeter and application defenses were sufficient before beginning significant deployments. He wanted to know how difficult it might be for an outsider to penetrate his security perimeter and access sensitive customer data, product development plans, or financial systems.

Barry asserted that his team had done a good job with security in the past. "What if you can't get in?" he asked rhetorically. Despite his confidence, his dull ache persisted. His nagging feeling compelled him to find out how good his defenses really were. He also wanted to get some benchmarks to see how well his company compared to other companies like his.

Barry wanted a McKinsey-style "diagnostic." This kind of diagnostic first states an overall hypothesis related to the business problem at hand and then marshals evidence (metrics) that supports or undermines the theory. The essence of the Mc­Kinsey diagnostic method is quite simple:

• The analysis team identifies an overall hypothesis to be supported. Example: "The firm is secure from wireless threats by outsiders."
• The team brainstorms additional subhypotheses that must hold for the overall hypothesis to be true. For example, to support the wireless hypothesis we just identified, we might pose these subhypotheses: "Open wireless access points are not accessible from outside the building" and "Wireless access points on the corporate LAN require session encryption and reliable user authentication."
• The team examines each subhypothesis to determine if it can be supported or disproved by measuring something. If it cannot, the hypothesis is either discarded or decomposed into lower-level hypotheses.
• For each lowest-level hypothesis, the team identifies specific diagnostic questions. The answers to the questions provide evidence for or against the hypothesis.

• Print

• Sample Questions For Finding Information Security Weaknesses
• A Few Good Information Security Metrics
• How to Use Metrics