Calling for Backup

Whenever I’m asked to do a security evaluation, the first question I ask is always the same: “Tell me about your backups.” The answers, of course, are all over the map. “Backups? We want you to evaluate our security,” is a typical response. Or, “We want to get all of our security set up first.” The most common response is something along the lines of, “We back up the important stuff, and we keep our fingers crossed about everything else.”

Of course, some organizations really have their acts together, with multilevel backups that take the data through two levels of spinning storage and then send it to tape, with every file logged in a central database. But those organizations are the exception, not the rule.

Why are backups on the bottom of so many to-do lists? In part, it’s because there is no glory in implementing and maintaining a backup system. Backups are an organizationally difficult task that, if everything goes right, should never need to be used.

Even when they work, they’re not the sort of thing that most IT professionals are eager to brag about. “Gee, Boss, the hackers broke in and zapped our accounts receivables database, but fortunately we were able to restore it from a backup!”

Nevertheless, if you are behind in your work and overextended in the budget department, there are many good reasons why you should be dropping other projects and working on your backup system instead.

The most important reason to back up your data should be obvious: If your security is so bad that some hacker does manage to break in and zap your accounts receivables database, at least you’ll be able to restore it from a backup.

But backups are about more than just protecting against hackers. Backups are good insurance against all kinds of disasters and mishaps, both natural and man-made.

Hourly backups let you quickly recover from something like an accidentally deleted file or a formatted volume. Weekly backups are vital for recovering important files that aren’t used very muchlike system configuration and startup files. Quarterly and yearly backups can be really useful in patent disputes and other kinds of litigation. These backups let you literally go back in time and retrieve memos and e-mail messages that have long since been otherwise deleted or lost.

As an information security professional, one of your main tasks is to be the information assurance person; that is, it’s your job to ensure that information is available when it’s needed. Backups play an important part in providing that assurance.

Of course, this ability to go back in time can be a double-edged sword. It’s great if your backup tapes have a downloaded copy of a webpage that shows conclusively how your competitor inadvertently disclosed details of a new invention a week before the company filed for a patent.

On the other hand, it can be particularly annoyingand potentially damagingwhen the attorney for a fired assistant requests the backup files of the CEO’s personal e-mail, fishing for proof that the boss really was having an affair with her just before she was terminated.

Indeed, because old e-mail messages can be so tremendously damaging, many organizations now have “document retention” policies that call for all e-mail (and all backup tapes) to be deleted after two or three years.

Because of such policies, a good backup system will make one kind of backup for quick system restores and another kind of separately labeled backup for long-term data recovery. That tape might have a copy of the database and configuration files, but not the e-mail. Disk to DiskOnce you’ve gone to the trouble of producing a backup tape, you must protect it. The first thing you should do is put it someplace safe (I know of one company that had its server stolen with the backup tape still in the drive!).

Organizations that use large tape libraries don’t need to worry about their hardware being stolenit’s too heavybut they still need to worry about the theft of a tape. What an easy way to steal corporate documents; just have a would-be janitor pocket a few backup tapes while he’s cleaning the computer room. You can make thefts less likely by putting a videocamera on your tape vault. Even better, have your backup software encrypt the backup tapes so that they will be useless to thieves. Just don’t forget the key. (I usually print out the key, seal it in an envelope and give it to the corporation’s attorney to keep in the company’s file.)

However, in these days when it costs less money to buy a 200GB hard drive ($150) than to purchase four tapes that can hold 50GB each ($200), a lot of organizations are giving up on tape for backup and just saving everything on spinning storagethat is, on hard disks. The problem with spinning disks, of course, is that they are prone to failure. For some organizations, that doesn’t matter a whole lot. In many cases, losing a backup is just not a big deal. Other organizations care a great deal about their backups and use the same sort of redundancy systems that they use for their primary storage.

These days, the standard way to protect against disk failure is with RAIDa redundant array of independent disks. The two most popular ways of configuring a RAID system is with so-called Level 1 mirroring and Level 5 parity systems. With Level 1, every byte of data is stored on two “mirrored” disks. If one fails, you’ve got the second. As a side benefit, since data has to be read from only one of the drives, RAID Level 1 systems can have twice the bandwidth for reading data back. Level 5 uses sets of three or more drives, and writes the data in such a way that if a drive fails, its data can be reconstructed from what’s on the remaining ones.

A lot of managers think that RAID reduces the need for a good backup system. It doesn’t. RAID protects you from a drive failure, but it doesn’t protect you from a computer virus or a hacker. Nevertheless, after spending a lot of money for disk systems, many managers are loathe to go out and purchase a suitable (and similarly expensive) backup system. Here’s one alternative that I’ve successfully used. If you have two reasonably similar computers, both with their own RAIDs, set the first computer to back itself up to the second, and make the second backup to the first.

On the other hand, many disk systems have the ability to make what’s called a snapshot, and this can be a substitute for some kinds of short-term backups. Snapshot This!A snapshot is a freeze-frame image of your computer’s hard drive. The location and contents of every file are noted in an instant, and then the computer continues with its work. Making a snapshot is like making an instant backup of the hard disk to the same drive.

Not every operating system supports snapshots, but the feature is becoming more popular. It was recently introduced in FreeBSD 5.0, for instance, although it wasn’t really reliable until the 5.2 version. Snapshots have been a part of NetApp’s gFiler appliances and EMC’s storage systems for years.

The advantage of the snapshot is that it can be made very fast and it takes up hardly any disk space at all. That’s because snapshots are implemented with a technique called “copy on write.” Basically, the operating system makes a map that notes the name and contents of every file. If an application tries to overwrite one of these files after the snapshot is made, the operating system writes the new file contents to an unused location of the hard drive and preserves the original contents.

The same thing is done with directories. If you try to delete a file inside a directory, the computer actually writes a second directory onto the disk that doesn’t have the file you just deleted. If you want to get back a file after you’ve accidentally deleted it, you just retrieve it from the snapshot.

On my primary server, for instance, I have a program that makes a snapshot every night at 11 p.m. I keep these snapshots for seven days, then they are automatically deleted.

The disadvantage of snapshots is that deleting a file doesn’t actually free up space on the diskthe blocks remain “used” until every snapshot that references the file is deleted too. And, of course, snapshots don’t protect you against a hardware failure or somebody accidentally formatting the hard drive.

One last thing: Once you have your backup system in place, you should practice trying to restore a backup from time-to-time. The best way to do this is to take a brand-new computer and a set of your backup tapes, and see if you can restore a 100 percent working system. Many organizations can’t, so don’t overlook this important test.