Security: Penetration Testing

Penetration tests are falling in popularity. Here are the keys to making them valuable again.

Maximum results also depend on getting quality service from the test provider. Not all consultants are created equal, of course. It's up to the CISO to make sure that he gets the best talent, and the best out of that talent. Some tricks of the trade are obvious: Check references and also resumes of those doing the testing. If the people named on the resumes aren't the ones in your lobby, send them home. Several experts recommend not using the same consultant twice in the same year—you want fresh eyes on your network. Ultimately, the goal is to receive a valuable and useful report, so CISOs should ask for samples of prior reports, several years' worth, if possible. The reports may sound similar, but if they're identical, that's quite possibly a sign of a consultancy that's simply going through the motions of point-and-click scanning. In order to provide a baseline for testing, CISOs recommend doing at least some in-house pen testing and vulnerability scans before contracting out the work.

They also insist on asking questions. If test providers say something is vulnerable, ask them if they were able to exploit it. If they say yes, ask them to show you how. If they can't reproduce the exploit, that may be a sign of a problematic test.

Another tactic is to use false positives. "I put 'Easter eggs' in there somewhere—false positives I know will be picked up by a particular scanning tool," says Ken Pfeil, a CSO turned consultant. Pfeil also changes a configuration here and there during testing and doesn't tell the consultants, to see if they catch it.

Of course, CISOs must stay engaged throughout the process. Don't just wait for the report to arrive before you start to think about what's happening with your pen test. Be involved in the meetings and watch some of the testing.

So while some CSOs may be grumbling about pen tests, it's clear that others want them. As a consultant, Pfeil says pen testing occupies most of his time. "Pen tests were a valuable tool in my life as a CSO, and they still are," he says. CISOs just need to apply these lessons to make sure they're getting the value they should.

Michael Fitzgerald is a freelance writer based outside Boston. E-mail feedback to Editor Derek Slater at dslater@cxo.com.