Software Patching: Patch and Pray

Patching-the only way to prevent poorly designed software from breaking everything-no longer works. And there's nothing you can do about it. Except maybe patch less. Or possibly patch more.

Hardly surprising, then, that philosophies on what to do next have bifurcated. Depending on whom you ask, it's either time to patch lessreplacing the process with vigorous best practices and a little bit of risk analysisor it's time to patch moreby automating the process with, yes, more software.

"We're between a rock and a hard place," says Bob Wynn, CISO of the state of Georgia. "No one can manage this effectively. I can't just automatically deploy a patch. And because the time it takes for a virus to spread is so compressed now, I don't have time to test them before I patch either."

With patching, the only certainty is that CISOs will bear the costs of bringing order to the intractable. In this penny-pinching era, other C-level executives are bound to ask the CISO why this is necessary, at which point someone's gonna have some 'splaining to do.The Learned ArtPatching is, by most accounts, as old as software itself. Unique among engineered artifacts, software is not beholden to the laws of physics in that it can endure fundamental change relatively easily even after it's been "built." Automobile engines don't take to piston redesigns post-manufacture nearly so well.

This unique element of software has contributed to (though is not solely responsible for) the software engineering culture, which generally regards quality and security as obstacles. An adage among programmers suggests that when it comes to software, you can pick only two of three: speed to market, number of features, level of quality. Programmer's egos are wrapped up in the first two; rarely do they pick the third (since, of course, software is so easily repaired later, by someone else).

Such an approach has never been more feckless. Software today is massive (Windows XP contains 45 million lines of code) and the rate of sloppy coding (10 to 20 errors per 1,000 lines of code) has led to thousands of vulnerabilities. CERT published 4,200 new vulnerabilities last yearthat's 3,000 more than it published three years ago. Meanwhile, software continues to find itself running evermore critical business functions, where its failure carries profound implications. In other words, right when quality should be getting better, it's getting exponentially worse.

Stitching patches into these complex systems, which sit within labyrinthine networks of similarly complex systems, makes it impossible to know if a patch will solve the problem it's meant to without creating unintended consequences. One patch, for example, worked fine for everyoneexcept the unlucky users who happened to have a certain Compaq system connected to a certain RAID array without certain updated drivers. In which case the patch knocked out the storage array.