Software Patching: Patch and Pray

Patching-the only way to prevent poorly designed software from breaking everything-no longer works. And there's nothing you can do about it. Except maybe patch less. Or possibly patch more.

Tim Rice, network systems analyst at Duke University, was one of the unlucky ones. "If you just jump in and apply patches, you get nailed," he says. "Patching is a learned art. You can set up six different systems the same way, apply the same patch to each, and get one system behaving differently."

Raleigh Burns, security administrator at St. Elizabeth's Medical Center, agrees. "Executives think this stuff has a Mickey Mouse GUI, but even chintzy patches are complicated."

The conventional wisdom is that when you implement a patch, you improve things. But Wynn isn't convinced. "We've all applied patches that put us out of service. Plenty of patches actually create more problemsthey just shift you from one vulnerability cycle to another," he says. "It's still consumer beware."

Yet for many who haven't dealt directly with patches, there's a sense that patches are simply click-and-fix. In reality, they're often patch-and-pray. At the very least, they require testing. Some financial institutions, says Shawn Hernan, team leader for vulnerability handling in the CERT Coordination Center at the Software Engineering Institute (SEI), mandate six weeks of regression testing before a patch goes live. Third-party vendors often take months after a patch is released to certify that it won't break their applications.

All of which makes the post-outbreak admonishing to "Patch more vigilantly" farcical and, probably to some, offensive. It's the complexity and fragility, not some inherent laziness or sloppy management, that explains why Slammer could wreak such havoc 185 days after Microsoft released a patch for it.

"We get hot fixes everyday, and we're loath to put them in," says Frank Clark, senior vice president and CIO of Covenant Health Care, whose six-hospital network was knocked out when Slammer hit, causing doctors to revert to paper-based care. "We believe it's safer to wait until the vendor certifies the hot fixes in a service pack."

On the other hand, if Clark had deployed every patch he was supposed to, nothing would have been different. He would have been knocked out just the same.Software Patching: Process HorribilisSlammer neatly demonstrates everything that's wrong with manufacturing software patches. It begins with disclosure of the vulnerability, which happened in the case of Slammer in July 2002, when Microsoft issued patch MS02-039. The patch steeled a file called ssnetlib.dll against buffer overflows.

"Disclosure basically gives hackers an attack map," says Gary McGraw, CTO of Cigital and the author of Building Secure Software. "Suddenly they know exactly where to go. If it's true that people don't patchand they don'tdisclosure helps mostly the hackers."