Operation clean sweep: How to disinfect a compromised network

You can't remove every bad scrap, but due diligence can go a long way toward yielding a clean, reliable network.

You either know your network is compromised or you're unaware your network is compromised. As far as I can determine, that's only a slight exaggeration. In the last seven years that I've concentrated on hacked networks -- most of which have been hit by advanced persistent threats -- I'm aware of only one company that has not been thoroughly and pervasively compromised.

Security professionals need to start by assuming their defenses have already been breached. It's hard to admit, but truth is always better than ignorance. It's like being mayor of a city and admitting that you probably have some crime despite a well-trained and appropriately funded police force. Crime happens. It's even in the Bible.

Once people reach the acceptance phase, they typically ask me this question: How can a company conduct a "clean sweep" to detect badness, root it out, and start over with a known clean environment?

The short answer: You can't. Well, you can, but to get a very high level of assurance would be very expensive and, in the end, probably not worth the money, time, and effort. It takes talent and time to gain a high level of confidence that you don't have any compromised computers requires exacting, high-detailed computer forensics.

Project superscrub

Most companies don't have images of the known, clean states of all their computers. If they did, they could do a quick comparison between clean images and current images. Even then, you'd have to sift through hundreds or thousands of little differences, from temporary files to legitimate updates to log files to any tiny object that changes during the normal course of business.

And without known, clean images? Forensic examiners must painstakingly review each computer and rule out thousands upon thousands of irregularities. The fastest I've seen any company turn this around is in about 24 hours for one computer. So if you want to do a clean sweep of your network with a high level of assurance, plug that into your resource calculator: 24 hours per computer per forensic investigator. Now you know why most companies don' t have the time or money.

Worse, determining if a particular computer is clean at a particular point in time doesn't prevent it from getting compromised immediately after. I've been involved in a few greenfield efforts, and most of the time those brand-new, clean environments end up compromised just like the old, legacy environments did. Why? Because the defenders didn't spend enough time ensuring that the old successful threats wouldn't be just as successful in the new environment. Simply moving users to new, clean computers doesn't mean they won't fall victim to the same old spearphishing attack.

Instead of striving for spic-and-span perfection, if you're determined to do a mop-up operation, I recommend a "limited assurance" clean sweep. This gives you reasonable assurance that your network is fairly clean and not completely owned by badness.

In a limited assurance audit, you select a sampling of computers in each role (such as file server, database server, Web server, client workstation, client laptop) to give you confidence over the whole population.

How many computers is enough?

A good rule of thumb is to sample at least two computers in each role at each major company site. If you want to sound impressive, you can calculate a statistical confidence level. (A great statistical calculator for figuring out sample sizes is located here.)

For example, if I had a population of 1,000 computers and wanted a confidence level of 90 percent plus or minus 15 percent (75 percent is still a fairly high level of assurance), I would need to sample 29 computers. If wanted a 5 percent confidence level instead of 15 percent, I would have to sample 214 computers.

Once you've determined your sample size, you can decide whether to do a full forensic analysis or a limited analysis for each computer. I can do a fairly good limited analysis in about one to two hours per computer. Here are the things I look for:

  • Up-to-date antimalware scanner (with a definition file no older than 24 hours) that is configured for constant detection
  • Up-to-date software and patches (no more than a week old)
  • Check security logs for abnormal events
  • Check all autostarting software and research any unknown software found
  • Review network traffic flows (in Windows you can do netstat -ano) looking for unusual activity
  • Check all installed software and make sure everything is legitimate and needed
  • Peruse folders and directories for rogue software or files
  • Look for files and folders with excessive permissions
  • Check the TCP/IP configuration and hosts file for rogue entries

In most cases, if a computer is compromised, these checks will catch it, depending on the talent of the person checking. Badness can always hide -- like rootkits do -- but normally, advanced persistent threats don't use rootkits, and the checks above will discover something funky that leads to even more specific discoveries.

I'm an even bigger fan of instituting checks that may detect badness even if your computers seem clean, including these:

  • Monitor net flows and look for strange or unusual network traffic flows
  • Use one or more honeypots
  • Use a whitelisting program in audit mode and look for unusual software execution

Most of us don't have the time to achieve perfect assurance. But that doesn't mean you can't do a fairly good assessment that will give you confidence about the security and safety of your network. Don't let perfect be the enemy of the good.

Copyright © 2014 IDG Communications, Inc.

The 10 most powerful cybersecurity companies