Steve Katz sat at his desk, reading an e-mail that he had hoped never to see. An outsider had access to the systems at his company. Katz, who was CISO at a large financial firm, would have to tell his boss. And that could be the start of something ugly.
The silver lining for Katz was this: The outsider was an ethical hacker Katz had hired to see if the company's systems could be penetrated. While it wouldn't be fun to deliver the news—"the guy had become a user of the system. He could've probably gotten access to critical applications," Katz says—at least it was just a penetration test.
"If you have significant value at risk, either your reputation or financial, a pen test is absolutely worth the price," Katz says. Lately, however, it seems that pen tests have fallen precipitously from the CISO radar. On the 2006 "CSO Magazine Sensor Survey," the tests were only the ninth priority for CISOs surveyed, down from third in 2005. While Katz says financial companies still use this tactic, the financial industry (which is usually on the high side of information security spending) is apparently bucking the trend. Anecdotally, CISOs elsewhere say they're tired of seeing reports listing scads of vulnerabilities that aren't legitimate, or of paying top dollar to have a consultant run a glorified system scan, or of too many security consultants with no understanding of how a corporate network really functions.
But guess what? Penetration tests still matter. In fact, Gartner Group earlier this year issued a report that pen tests are more important now than ever before, because hackers have shifted from mass attacks like worms to targeted, multipronged attacks on specific companies. A well-executed penetration test can identify the most critical holes in an organization's defensive net—including the holes exploited by social engineering. CISOs who swear by these tests say you just need to sharpen your approach to them to make them useful, and here they offer tips on how to do just that.
Penetration Tests: Failing Grades
There are plenty of reasons why pen tests seem to have lost their ink.
For one thing, the results usually surprise no one: The network is vulnerable. One former security consultant who goes by the handle Hellnbak (and who now works at eEye Digital Security) said in an e-mail, "I've done hundreds of pen tests, and I was able to break into the network every time, with two exceptions. For the most part, companies should be taking the money they are wasting on a pen test and spending it on a secure network design session."
Another reason is that most companies can't afford an open-ended penetration test, so they set time limits—which real hackers don't have, notes Mark Weatherford, CISO for the state of Colorado. Also, if the network is already compromised, a pen test might not find out. "I love to go into companies and put a sniffer down on the inside and see how many systems are already compromised, what spyware is already on there," says Peiter "Mudge" Zatko, a longtime security researcher who is now a division scientist at BBN Technologies. "Pen test games don't even address this. They're just trying to get in. If a system is already compromised it negates any work you've done on the perimeter."
There are myriad other reasons people say they dislike pen tests—the cost, the potential for disrupting your business if the pen test takes down a system, the chance that a jaded consultant might just run a few scanning tools looking for known security holes and call it a pen test. John Pescatore, a security analyst at Gartner, thinks that pen tests fell out of favor in part because of what he calls "gray-hat" hackers, college kids who were doing work so cheaply that seasoned professionals got out of the business. Pescatore says he was aware of kids charging $500 for a test—compare that to one consultant interviewed for this story, who says she charges $500 per target IP address, with a minimum of five addresses. Pescatore argues that the kids didn't really know how to construct exercises that would reflect the complexities of corporate networks, and those low-cost, low-quality tests soured many CSOs on the concept.
Practical Matters
The first step in making penetration tests valuable is to understand how they should fit into the information security arsenal. Clearly, no one should rely on penetration tests as the only answer. Pescatore cautions that legitimate tests are too costly and time-intensive to do more than once a quarter. And any change to the system can render the previous pen test moot. "It's a snapshot in time," says Carole Fennelly, who runs the security consultancy WizardKeys. She says that pen tests are best used as a way to get an extra set of eyes on a network after major system upgrades.
Vulnerability assessments are a great companion piece to penetration tests. The difference between the two is critical, and André Gold, director of information security at Continental Airlines, can explain that difference.
"I have hundreds of locations with thousands of hosts, and a lot have bandwidth that's sub-128K—there is no way on God's green Earth that I could do a pen test across all these hosts," Gold says. But he can do a vulnerability assessment of them, and then compare vulnerabilities with the known potential dangers. Some vulnerabilities simply won't be problems, he says. An unprotected 56k modem sitting in an airport is one example. It's a vulnerability, and should be identified as such in a good vulnerability assessment. However, the low bandwidth may make that modem a rotten launchpad for attacking the network. And there may be additional controls on the network that further protect against exploitation of that modem. Gold certainly has to know what those controls are, and a penetration test can help determine whether the controls are in fact preventing the vulnerability from actually endangering the network. That type of information can help a CISO prioritize his work. A vulnerability that is covered by internal controls or other defenses may move lower on the fix-it list than one that is proven accessible through an outside penetration test.
In Gold's nearly 10 years at Continental, the first seven as director of Internet services, he's seen scores of penetration tests. The methodology, he says, remains similar to what it was in 1996 (though it has naturally expanded to include such new protocols as SOAP and variants of XML). But the process and the tools have changed—the tests don't take as long, in part because the tools are more automated. In 1996, DNS searches, server pings and manual telnetting were all common parts of information gathering during a penetration test, and now these are handled by tools like Nmap, Gold says. He also says vulnerability scanners can find potential configuration errors and buffer overruns, which was not the case in the late '90s. There are even commercial-grade exploit tools like Core Impact, which Continental uses internally, particularly after installing new application releases. Continental used to outsource a broad range of work, including audits, vulnerability assessments and pen tests. Now, the company does much of this work in-house, security working together with the internal audit group.
Gold's bottom line on pen tests: Audits and assessments are well and good. Companies need such tools. But they can't prove whether a vulnerability equals a corporate liability, where a pen test can. A good pen test, then, provides peace of mind.
Sharpening the Pen
Once a CISO knows where pen tests fit into the overall security scheme, he can move on to the practicals of getting the most of out them.
For starters, keep an eye on business activities and requirements. "You don't want to do a pen test to the tax agency on April 15," says Will Pelgrin, director of the New York State Office of Cybersecurity and Critical Infrastructure Coordination. New York's agencies handle their own penetration tests now, but Pelgrin also is setting up pen tests as a service his group would offer via third-party firms. He's doing this now in part because of timing—his office has finished a series of security to-dos he'd deemed more important, such as a statewide security policy, mitigation gap analysis, compliance policies and the like. He's also looking to give the agencies a baseline for how to do the tests and some recommendations on people to perform them.
In offering the service, Pelgrin emphasizes the need for cooperation with the pen test providers and the agencies being tested. Poorly timed or poorly planned penetration tests can do more harm than good, he says, a concern borne out by one consultant who remembers when a financial services IT director scheduled a denial-of-service attack for the close of trading, and nearly shut down the company's systems. "Nobody would tell us it was a test, and we almost called the FBI; think of the embarrassment that would have caused the firm," the consultant says.
Colorado's Weatherford notes another key to valuable tests: Don't use them until you've made a reasonable effort to get your network secure. He's seen some people guilty of thinking a penetration test is the security version of pushing the Staples Easy Button. "People think you push the easy button and it will happen, your problems are clear." In fact, he says, the organization that does not have mature controls in place around IT systems will find pen tests of little value. "It just points out that your system can be exploited. Big deal." In fact, he notes that this type of exercise can be damaging to poorly built systems, which is another reason to use pen tests with care. "I consider a pen test to be the supreme test for a mature organization. It's important to remember that pen tests are invasive activities and can break things," Weatherford says.
Echoing Gold's approach at Continental, Weatherford says the run-up to pen testing is to first establish policies, then conduct vulnerability assessments to identify weaknesses, and then remedy or mitigate the key weaknesses. Only then will the pen test yield maximum bang for the buck.
Sharper Still
Maximum results also depend on getting quality service from the test provider. Not all consultants are created equal, of course. It's up to the CISO to make sure that he gets the best talent, and the best out of that talent. Some tricks of the trade are obvious: Check references and also resumes of those doing the testing. If the people named on the resumes aren't the ones in your lobby, send them home. Several experts recommend not using the same consultant twice in the same year—you want fresh eyes on your network. Ultimately, the goal is to receive a valuable and useful report, so CISOs should ask for samples of prior reports, several years' worth, if possible. The reports may sound similar, but if they're identical, that's quite possibly a sign of a consultancy that's simply going through the motions of point-and-click scanning. In order to provide a baseline for testing, CISOs recommend doing at least some in-house pen testing and vulnerability scans before contracting out the work.
They also insist on asking questions. If test providers say something is vulnerable, ask them if they were able to exploit it. If they say yes, ask them to show you how. If they can't reproduce the exploit, that may be a sign of a problematic test.
Another tactic is to use false positives. "I put 'Easter eggs' in there somewhere—false positives I know will be picked up by a particular scanning tool," says Ken Pfeil, a CSO turned consultant. Pfeil also changes a configuration here and there during testing and doesn't tell the consultants, to see if they catch it.
Of course, CISOs must stay engaged throughout the process. Don't just wait for the report to arrive before you start to think about what's happening with your pen test. Be involved in the meetings and watch some of the testing.
So while some CSOs may be grumbling about pen tests, it's clear that others want them. As a consultant, Pfeil says pen testing occupies most of his time. "Pen tests were a valuable tool in my life as a CSO, and they still are," he says. CISOs just need to apply these lessons to make sure they're getting the value they should.
Michael Fitzgerald is a freelance writer based outside Boston. E-mail feedback to Editor Derek Slater at dslater@cxo.com.