How to get rid of advanced persistent threats

Having been involved in fighting off nearly two dozen APT (advanced persistent threat) attacks over the past three years, I’m somewhat experienced at eradicating them — or, more accurately, minimizing them — in large networks. This type of attack isn’t impossible to detect; in fact, that’s the easy part. Advanced persistent threats are, however, exceedingly difficult to remove from your network without severely disrupting revenue-generating operations and/or exposing your environment to additional compromises.

Although every instance of an advanced persistent threat is unique, I can offer general suggestions for facing such threats for the first time. Finding and eliminating — or at least reducing — an APT attack requires careful and stealthy planning, so as not to alert the attackers to your defensive maneuvers and give them a chance to counter your efforts.

Preparing your network and your staff for remediation day If you’re an IT admin, communicate the known extent of the problem and initial plans for dealing with the advanced persistent threat to IT senior management. This will often morph into presentations to overall senior management, likely to the board of directors, regulators, partners, vendors, and so on. Let senior management dictate who gets to know what and when.

The first major technical response should be to implement more detection across your network; you need to find out the severity of the APT problem. Which computers are owned? Are passwords known? What tools and malware are being used? Is email compromised? Where is the data flowing to, both internally and externally? At a minimum, detecting APT usually means implementing host and network intrusion detection software if it is not already in use.

Next, you need to determine the best way to handle the problem. You might choose to remove each compromised computer from the network immediately. Alternatively, you might initially allow those systems to continue running unabated to prevent the APT planners from becoming aware that they’ve been discovered. This is an individual risk decision for each company; I’ve seen it handled both ways.

From there, invite remediation participants and make an eradication plan. Your network security team should include technical staff, senior management representatives, vendor specialists, APT specialists, affected business unit team leaders, messaging groups, project managers, and whoever else needs to be involved. In general, start small, and bring in people as necessary. Everyone involved needs to sign an NDA document, even if the company already has one. You want to reinforce the seriousness of keeping the information secret from the large entity until a formal communication plan can be created and implemented.

Assign the APT and remediation event a keyword that all participants will use in online communications. Use phrases such as “health care update,” “baseball game,” or “travel policy.” You want something that should be innocuous enough not to attract unnecessary attention from your APT attackers.

Getting rid of APT isn’t that hard, but doing so without causing operational interruption is the difficult part. To that end, inventory all applications and services before mounting your mass-cleansing remediation event across the network. Assign ownership — that is, determine who is responsible for answering questions about each resource, as well as keeping it up and running. Document what user and service accounts are necessary to remain functional.

In addition, assign criticality: Which apps and services must remain active with the least amount of downtime? What is the worst-case scenario acceptable to senior management? In one recent instance, the unacceptable event was the late filing of public financial statements, but everything up to that point was considered reasonable. Other companies define acceptable downtime of their mission-critical apps in hours.

Next, inventory users, computers, service accounts, network devices, and Internet connection points. How many do you have, and where are they? Develop lifecycle management policies and procedures around all of them, from creation and ownership to deletion after they’re no longer needed.

Most environments have too many objects, a lack of clear ownership, and a general inability to determine what is justified or needed among existing items. Break the cycle. No action is as secure as removing an object you don’t need. For example, companies often end up significantly reducing the number of elevated accounts in their environments. It’s great to pare down to the bare minimum during remediation, but how do you keep it that way over the long term?

Last, before remediation day, make sure patching is up-to-date. This can be done ahead of time and is beneficial for many reasons beyond getting rid of an APT. Conduct health checks of your network, your WAN, and your most important infrastructure systems. You want the network and environment operating at top efficiency before you try to push big changes on remediation day.

Game day: Hit the attackers fast and hard Remediation day should be planned far ahead of time. Have a defined, well-tested set of steps, and plot out timelines and responsibilities. Everyone should know what they are doing and when. At the very least, remediation days usually start with disconnecting the company’s network from the Internet so that APT attackers cannot respond to and control what is going on.

Bring all known APT systems offline and completely rebuild. Change all account passwords, including service accounts. Consider requiring two-factor authentication for elevated account use. Test all mission-critical applications and services with the new authentication credentials. Some companies go so far as to completely rebuild their LDAP/Active Directory infrastructures, which is really the only way to significantly minimize APT risk and re-exploitation.

Some companies cut over to a new infrastructure and others migrate over time; the former approach is more secure than the latter, but the latter is easier and smoother for operations.

Require that all users receive education about advanced persistent threats and current malware tricks (such as malformed PDFs, fake antivirus, social engineering) before being allowed back on the network with their new passwords. Employees can be informed of the APT event or simply told that the new passwords are part of the company’s renewed effort to minimize security risk, depending on what the communication team decides.

Don’t let your guard down against APT Employees and management should expect APT attackers to fight back with a vengeance to re-establish their old footholds. The first few days after remediation are often the highest risk.

I’m a big fan of computer and domain isolation. Most workstations don’t need to talk to other workstations, and most servers don’t need to talk to other servers. Define what communication pathways are needed and block the rest. Use the fastest and dumbest device/service to accomplish the task. Use intelligent (but slower) application-level firewalls and proxies only where they are needed.

Make sure internal development teams are practicing secure development lifecycle techniques. Additionally, implement comprehensive event log management systems, detection, and response. Most malicious behavior would have been noticed if the event logs were configured correctly and reviewed.

In the future, look for unusual network traffic patterns. This is often the first easy-to-see sign of an APT attacker. It’s what they do: Steal information and transfer it to places where you normally don’t send data.

Finally, consider implementing one or more early-warning honeypots. They are low cost, low noise, and among the best detection devices for any network. In order for a hacker to exploit a network, they have to touch computers. Honeypots are nonproduction assets and, as such, should never be touched after the initial fine-tuning.

In closing, minimizing and eradicating advanced persistent threats is among the hardest challenges any company can face. It can be difficult, if not impossible, to completely wipe out these invaders if you’re constrained by senior management, operational directives, and financial considerations. For many companies, the new normal is living with the risk of advanced persistent threats forever, but any company can fight the good fight.