Pen Testing: Dead in 2009

Penetration testing: Security experts mention it all the time as one of the essential tools of defense-in-depth. Companies have raked in the dough selling the service and the tools for years.

But is it possible that penetration testing — the art of probing company networks in search of exploitable security holes that can then be fixed — is an idea whose time is about to expire?

If you ask Brian Chess, co-founder and chief scientist of business software assurance (BSA) vendor Fortify Software Inc., the answer is yes.

“Death sounds rather gloomy, but stuff in high tech dies all the time,” Chess said in an interview Tuesday. “Desktop publishing? Dead — but not gone. Personal Digital Assistant (PDA)? Many of the concepts are still with us, but the PDA is dead.”

Penetration testing is headed for a similar fate, he said. The concept as we know it is on its death bed, waiting to die and come back as something else. That doesn’t mean pen testers will suddenly be unemployed, he said. It’s just that they “won’t be as cool” as they’ve been in more recent years.

Customers are clamoring more for preventative tools than tools that simply find the weaknesses that already exist, he said. They want to prevent holes from opening in the first place.

“Death doesn’t mean it goes away, it means it transforms. Pen testing will be reborn in the area of production monitoring and measurement,” Chess said. “The goal won’t be that failure is found and must be fixed. The goal is that failures will become a much rarer event.”

Pen testing has its fans

Naturally, security practitioners who swear by pen testing as a critical component of a layered security program are reacting to his hypothesis with more than a little skepticism.

Jennifer Jabbusch, CISO at Carolina Advanced Digital Inc. in the Raleigh-Durham area of North Carolina, took issue with Chess’ basic premise that penetration testing will become a component of monitoring and measuring.

“Pen testing will continue,” she said in an exchange over the Twitter social networking site. “Monitoring and measuring is not pen testing. It’s what you do after pen testing.”

She also faulted the example of desktop publishing being a dead art, saying, “Desktop publishing isn’t dead. In fact, it’s grown. Now you can design on your desktop and deliver via the Internet for printing at FedEx/Kinkos.”

Others agree penetration will continue, but don’t necessarily think Chess’ position is all that off the mark.

Max Caceres, director of research and development at Matasano Security in New York, said he can understand the perspective of people who want penetration testing to be part of something larger.

“I can totally see where his customers are coming from,” Caceres said. “All things being equal, preventing holes from even existing is a much more interesting approach than riding the find-report-hope-somebody-fixes-it hamster wheel.”

But, he added, Chess’ prediction may be more of an imagined utopia than a real alternative.

“We have been findings bugs for a while, the most common problems are well understood and documented, yet we keep deploying vulnerable apps,” he said. “If we believe true perfection is unattainable — and I do, particularly for application development, we have yet to invent the tool that produces bug-free code — then apps will always have bugs that need fixing, and some of them will be security related.”

And that’s where penetration testing will remain valuable, he said.

Kevin Riggins, a senior information security analyst for a company in the Des Moines, Iowa, area, said it’s hard to argue with Chess’ premise that the goal should be fewer failures. But he doesn’t believe that sentiment has anything to do with the need for or the use of penetration testing. Furthermore, he said, echoing Jabbusch, production monitoring and measuring and penetration testing do not address the same issue.

“The first measures the availability and effectiveness of your production environment,” he said in exchanges via Twitter and e-mail. “The second measures its ability to resist intrusion or attack. They are not the same and you can’t get from one to the other by transformation.”

A better argument for the death of penetration testing is that there will always be issues found, some of which can not be fixed or effectively mitigated, he added. Therefore, what is the real value to the organization in performing this type of test?

“Don’t get me wrong, I don’t subscribe to this argument either,” Riggins said.

In the final analysis, he said, security pros can’t stop performing penetration tests until the current compliance requirements are removed. That’s not happening any time soon.

“Penetration tests and vulnerability scans help us find where our processes, procedures, and standards might need work,” he said.