Intrusion Detection: Ain't No Flyswatter Big Enough

What do you do when somebody breaks into one of your organization's servers? When waving your hands wildly doesn't help, you'll need an intrusion detection plan.

By Simson Garfinkel

November 01, 2004 — CSO — I first started seeing this problem back in the late 1980s. Hackers would break into a system, set up a few accounts of their own, and then start installing Trojan horses and other kinds of back doors so that they could always return. Trying to contain the damage was like swatting at flies. The system management might remove the bad guy accounts, but they would come back through the back doors and recreate them. Management might reinstall the operating system, but the bad guys would still come back—perhaps through some new service that had been left behind. And since the bad guys invariably had root-level access, there was always a chance that they would mount a revenge attack and wipe out the system.

There was really only one way to prevent systems like this from spiraling out of control: save whatever you could, then log out and "nuke from orbit. In other words, a standard operating procedure was to copy critical files and data off the system, then reformat its hard drive and do a clean installation.

The "nuke from orbit" approach works, but like real-life atomic weapons, most incidents don't need such drastic measures. It's incredibly wasteful to spend a week or more reinstalling a computer's operating system and applications because of a few modified files. During that time, you are either running a compromised machine or your server is down.

What's even worse, nuking occasionally doesn't work. Yes, a complete reinstall would get rid of the compromised log-in program and mail server that the bad guys left behind, but reinstalling the operating system doesn't do much good if the original penetration was due to a stolen password.

Into this mess came what should properly be considered one of the computing industry's first intrusion detection systems. Called Tripwire, the program took a snapshot of the computer's operating system, recording the modification date and a cryptographic checksum for every program on a given computer system. This database would then be stored not on the computer itself, but on removable media—usually a floppy disk. If you discovered that your machine was compromised, you could run Tripwire a second time and compare the two databases to find which files the bad guys had modified. Even better, you could run Tripwire on a regular basis to find out if your machine was compromised.

A graduate student named Gene Kim and his adviser, Gene Spafford, developed Tripwire at Purdue University. (Professor Spafford and I are coauthors on several computer security books.) Realizing he had a winner, Kim took the research and cofounded a company, also called Tripwire, based in Portland, Ore.

• Print