Software Patching: Patch and Pray

Patching-the only way to prevent poorly designed software from breaking everything-no longer works. And there's nothing you can do about it. Except maybe patch less. Or possibly patch more.

Essentially, disclosure's a starter's gun. Once it goes off, it's a footrace between hackers (who now know what file to exploit) and everyone else (who must all patch their systems successfully). The good guys never win this race. Someone probably started working on a worm into ssnetlib.dll when Microsoft released MS02-039, or shortly thereafter.

In the case of Slammer, Microsoft built three more patches in 2002MS02-043 in August, MS02-056 in early October and MS02-061 in mid-Octoberfor related SQL Server vulnerabilities. MS02-056 updated ssnetlib.dll to a newer version; otherwise, all of the patches played together nicely.

Then, on October 30, Microsoft released Q317748, a nonsecurity hot fix for SQL Server. Q317748 repaired a performance-degrading memory leak. But the team that built it had used an old, vulnerable version of ssnetlib.dll. When Q317748 was installed, it could overwrite the secure version of the file and thus make that server as vulnerable to a worm like Slammer as one that had never been patched.

"As bad as software can be, at least when a company develops a product, it looks at it holistically," says SEI's Hernan. "It's given the attention of senior developers and architects, and if quality metrics exist, that's when they're used."

And then there are patches.

Patch writing is appropriated to entry-level maintenance programmers, says Hernan. They fix problems where they're found. They have no authority to look for recurrences or to audit code. And the patch coders face severe time constraintsremember there's a footrace on. They don't have time to communicate with other groups writing other patches that might conflict with theirs. (Not that they're set up to communicate. Russ Cooper, who manages NTBugtraq, the Windows vulnerability mailing list, says companies often divide maintenance by product group and let them develop their own tools and strategies for patching.) There's little, if any, testing of patches by the vendors that create them.

Ironically, maintenance programmers write patches using the same software development methodologies employed to create the insecure, buggy code they ostensibly set out to fix. Imagine that 10 people are taught to swim improperly, and one guy goes in the water and starts to drown. Do you want to rely on the other nine to jump in and save him?

From this patch factory comes a poorly written product that can break as much as it fixes. For example, an esoteric flaw found last summer in an encryption programone so arcane it might never have been exploitedwas patched. The patch itself had a gaping buffer overflow written into it, and that was quickly exploited, says Hernan. In another case last April, Microsoft released patch MS03-013 to fix a serious vulnerability in Windows XP. On some systems, it also degraded performance, by roughly 90 percent. The performance degradation required another patch, which wasn't released for a month.