Software Patching: Patch and Pray

Patching-the only way to prevent poorly designed software from breaking everything-no longer works. And there's nothing you can do about it. Except maybe patch less. Or possibly patch more.

1 2 Page 2
Page 2 of 2

Then again, companies take more than a week to stick a service pack into a network. After all, single patches require regression testing and service packs are hundreds of security patches, quality fixes and feature upgrades rolled together. In a crisis, upgrading a service pack that was days old wasn't reasonable. Cooper soon learned that Best Software's MAS 500 accounting software wouldn't run with Service Pack 3. MAS 500 users who installed SP3 to defend against Slammer had their applications fall over. They would have to start over and reformat their machines. All the while everyone was trying to beat Slammer to the workweek to avoid a severe uptick in Slammer infections when millions of machines worldwide were turned on or otherwise exposed to the worm that, over the weekend, remained blissfully dormant.

"By late Sunday afternoon, Microsoft had two rooms set up on campus," says Cooper. "Services guys are in one room figuring out what to say to customers. A security response team is in the other room trying to figure out how to repackage the patches and do technical damage control.

"I'm on a cell phone, and there's a guy there running me between the two rooms." Cooper laughs at the thought of it.Repeat MistakesAs the volume and complexity of software increases, so does the volume and complexity of patches. The problem with this, says SEI's Hernan, is that there's nothing standard about the patch infrastructure or managing the onslaught of patches.

There are no standard naming conventions for patches; vulnerability disclosure comes from whatever competitive vendor can get the news out there first (which creates another issue around whether vendors are hyping minor vulnerabilities in order to associate themselves with the discovery of a vulnerabilityyet another story for another day). Distribution might be automated or manual; and installation could be a double-click .exe file or a manual process.

Microsoft alone uses a hierarchy of eight different patching mechanisms (the company says it wants to reduce that number). But that only adds to more customer confusion.

"How do I know when I need to reapply a security roll-up patch? Do I then need to reapply Win2K Service Pack 2? Do I need to re-install hot fixes after more recent SPs?" Similar questions were posed to a third-party services company in a security newsletter. The answer was a page-and-a-half long.

There's also markedly little record-keeping or archiving around patches, leaving vendors to make the same mistakes over and over without building up knowledge about when and where vulnerabilities arise and how to avoid them. For example, Apple's Safari Web browser contained a significant security flaw in the way it validated certificates using SSL encryption, which required a patch. Every browser ever built before Safari, Hernan says, had contained the same flaw.

"I'd like to think there's a way to improve the process here," says Mykolas Rambus, CIO of financial services company WP Carey. "It would take an industry bodya nonprofit consortium-type setupto create standard naming conventions, to production test an insane number of these things, and to keep a database of knowledge on the patches so I could look up what other companies like mine did with their patching and what happened."

Rambus doesn't sound hopeful.

There won't be a formal announcement of the fact, and no one really planned it this way, but Slammer has become something of a turning point. The fury of its 10-minute conflagration and the ensuing comedy of a gaggle of firefighters untangling their hoses, rushing to the scene and finding that the building burnt down left enough of an impression to convince many that patching, as currently practiced, really doesn't work.

"Something has to happen," says Rambus. "There's going to be a backlash if it doesn't improve. I'd suggest that this patching problem is the responsibility of the vendors, and the costs are being taken on by the customers."

There's good news and bad news for Rambus. The good news is that vendors are motivated to try and fix the patch process. And they're earnestone might say even religiousabout their competing approaches. And the fervent search for a cure has intensified markedly since Slammer.

The bad news is that it's not clear either approach will work. And even if one does, none of what's happening changes the economics of patching. Customers still pay.More or LessThere are two emerging and opposite patch philosophies: Either patch more, or patch less.

Vendors in the Patch More school have, almost overnight, created an entirely new class of software called patch management software. The term means different things to different people (already one vendor has concocted a spinoff, "virtual patch management"), but in general, PM automates the process of finding, downloading and applying patches. Patch More adherents believe patching isn't the problem, but that manual patching is. Perfunctory checks for updates and automated deployment, checks for conflicts, roll back capabilities (in case there is a conflict) will, under the Patch More school of thought, fix patching. PM software can keep machines as up-to-date as possible without the possibility of human error.

The CISO at a major convenience store retail chain says it's already working. "Patching was spiralling out of control until recently," he says. "Before, we knew we had a problem because of the sheer volume of patches. We knew we were exposed in a handful of places. The update services coming now from Microsoft, though, have made the situation an order of magnitude better."

Duke University's Rice tested patch management software on 550 machines. When the application told him he needed 10,000 patches, he wasn't sure if that was a good thing. "Obviously, it's powerful, but automation leaves you open to automatically putting in buggy patches." Rice might be thinking of the patch that crashed his storage array on a Compaq server. "I need automation to deploy patches," he says. "I do not want automated patch management."

The Patch Less constituency is best represented by Peter Tippett, vice chairman and CTO of TruSecure. Tippett is fanatical about patching's failure. Based on 12 years of actuarial data, he says that only about 2 percent of vulnerabilities result in attacks. Therefore, most patches aren't worth applying. In risk management terms, they're at best superfluous and, at worst, a significant additional risk.

Instead, Tippett says, improve your security policylock down ports such as 1434 that really had no reason to be openand pay third parties to figure out which patches are necessary and which ones you can ignore. "More than half of Microsoft's 72 major vulnerabilities last year will never affect anyone ever," says Tippett. "With patching, we're picking the worst possible risk-reduction model there is."

Tippett is at once professorial and constantly selling his own company's ability to provide the services that make patching less viable. But many thoughtful security leaders think Tippett's approach is as flawed and dangerous as automated patch management.

"There's no place for that kind of thinking, to patch less," says St. Elizabeth's Burns. "As soon as an exploit takes advantage of an unknown vulnerabilityand one willthose guys will be scratching their heads. He's using old-school risk analysis. How can you come up with an accurate probability matrix on blended threat viruses using 12 years of data when they've only been around for two years?"

Add to this a sort of emotional inability to not patchsort of like forgetting to put on your watch and feeling naked all day. Several CISOs described an illogical pull to patch, even if the risk equation determined that less patching is equally or even more effective.

There's also an emerging hybrid approachwhich combines the patch management software with expertise and policy management. It also combines the costs of paying smart people to know your risks while also investing in new software.

"There's a huge push right when P&L captains are telling CISOs to keep costs down," says Hernan. That might explain why the executive security ranks are far less enamored by the Patch Less/Patch More philosophies. The polar approaches haven't yet spurred CISOs to take sides so much as they've flummoxed them. Ambivalent confusion reigns.

Hernan says, "I can understand the frustration that can lead to the attitude of, 'Forget it, I can't patch everything,' but that person's taking a big chance. On the other hand, he's also taking a big chance applying a patch."

"I don't have much faith in automated patching schemes," says Rambus. "But I could be convinced."

Georgia's Wynn is ambivalent too. "If you think patch management is a cure, you're mistaken. Think of it as an incremental improvement. I have to take a theory of the middle range," he says vaguely. PostscriptOn Monday after Slammer hit, Microsoft re-released MS02-061 to cover up the memory leak and update ssnetlib.dll, and it was much easier to install. Of course, by then, Slammer was already pandemic. Microsoft itself was infected badly, prompting a moment of schadenfreude for many. ISP networks had collapsed; several root DNS servers were overwhelmed; airlines had canceled flights; ATM machines refused to hand out money. In Canada, a national election was delayed.

And after all that, the patches had, at best, a miniscule mitigating effect against Slammer. What ended up preventing Slammer from worming its way into the workweek and causing even more damage, it turns out, was a rare and unusual gesture by ISPs. That same Monday, they agreed to cooperatively block Internet traffic on UDP port 1434, the one Slammer used to propagate itself. "That's what allowed us to survive," says Cooper.

And surely, with ISPs blocking the door, companies would seize the opportunity to update, test and deploy the new patches. Or, if they felt up to it, they could upgrade to Service Pack 3. They could use the time to locate and patch all of their MSDE clients and, once and for all, kill Slammer dead.

Ten days later, when ISPs opened port 1434 again, sure enough, there was a spike in Slammer infections of SQL Servers. Six months later, in mid-July, as this story went to press, the Wormwatch.org listener service showed Slammer remained the most prevalent worm in the wild, twice as common as any other worm. It was still trolling for, and finding, unpatched systems to infect.

Copyright © 2003 IDG Communications, Inc.

1 2 Page 2
Page 2 of 2
7 hot cybersecurity trends (and 2 going cold)