Understanding incident response: 5 tips to make IR work for you

LAS VEGAS (Black Hat) — IT professionals, security experts, and researchers have traveled to Sin City this week, in order to attend the annual Black Hat security conference. While many of the presentations and demos at Black Hat will focus on trends and the latest technological advances for those who attack the network and those who defend it, there isn’t much on the topic of incident response.

In the grand scheme of things, though it is marketed, debated, and packaged as its own layer of proactive defense, incident response could arguably be called disaster recovery, or business continuity planning. Once an incident has been discovered — either as it happening or long after the fact — steps must be taken to address it, ensure the organization recovers from it, and that it doesn’t happen again, but before this can happen a response plan needs to be in place.

[More from Black Hat: General Alexander heckled during keynote address]

On paper, this sounds like an easy thing to do, but the experts and trench workers that CSO spoke to this week at Black Hat were quick to note that this simply isn’t true. Incident response is something that is developed and something that changes with the organization over time. Incidents can be technical or physical, and while you can’t prepare for everything, it’s wise to at least prepare for the most likely threats that your organization will face.

“Companies have figured out that they can invest all they want into defensive strategies, but at some point in time they will fail. Time and time again, that has been proven. Even the best of systems fail, because the longer theyre out there, the more the attackers will learn how to circumvent them,” said Ken Silva, the President Cyber Strategy for ManTech Cyber Solutions International.

“So they spend all this time, and all this training, and all this education that they’ve got, and all the money that they invested in perimeter defense, and even internal defenses, but they didn’t spend a dime on incident response. Incident response tends to be, in most cases, an ad-hoc thing that’s put together as needed; it’s almost like a volunteer fire department. The only difference is that the volunteer fire department is properly trained, they have the right processes, [and] they have the right tools.”

Developing an incident response plan, and ensuring that it aligns to the organization’s goals and needs, as well as existing policy and compliance regulations, can be a daunting. Moreover, the process will require all sides of the business to communicate, which in itself can be quite the task. As one security worker at Black Hat put it, “it’s like herding cats.”

At the very least, the legal department needs to be involved in order to ensure regulatory and compliance needs are met, business leaders (i.e. everyone at the C-Level within the organization) will each need to ensure that they’ve got a say in things, and then IT needs to ensure the plan is maintained and updated regularly.

Eric Fiterman, the founder and CEO of Spotkick, a SaaS platform for security analytics and intelligence, told CSO during an interview that incident response will often depend on the size of the organization itself. His point being that no two plans are alike.

“Because some enterprises and organizations are large enough that they can spend time tracking things down; and then there are a lot of IT departments where that’s not the main concern. The main concern is making things reliable and getting systems recovered and up as quickly as possible,” he said.

The key, Fiterman added, is that organizations understand that incidents will happen.

“You’re going to have some type of malicious activity that’s going to impact what you’re doing,” he said, so organizations need to have a plan in place to deal with the various circumstances as they arise.

[Related: Fatal half measures in incident response]

According to the Black Hat attendees that CSO spoke with, there are a few common problems when it comes to incident response, and unless plans were developed and tested beforehand, then these common problems show themselves at the worst possible time; during an actual incident.

“What IT gets right is that they know their infrastructure. They know where their data of value is, they know ingress points [and] they know egress points. It’s their network, they understand how it works. What they get wrong is they don’t use the working knowledge that they have of the network to understand how and incident would occur,” Chris Pogue, the Director of Trustwave’s SpiderLabs, told CSO in an interview.

“Attackers are there for a reason, not for fun and games. They’re there to steal something, so that they can extract it and monetize it. So if [IT] would integrate the knowledge that they already have into their incident response capabilities, they’d be so much better off; because they’d know the targets, they could defend the targets, [and] they would know their ingress points, and their egress points.”

One of the often repeated problems with incident response is that organizations rarely understand those who are attacking them, what the attacker is looking for, and how they are trying to get it.

This means understanding what’s valuable to an attacker, and where it lives on your network. As mentioned by Pogue, knowing all the routes and access points to the critical data is a must, so that when something happens you can accurately flag the incident and deal with it appropriately.

The key though, is to apply this line of thinking to incidents other than vulnerability exploitations and malware, because there needs to be plans and processes in place to deal with lost or stolen laptops, accidental data disclosures, and malicious staffers who abuse their access. Further, there should be plans in place to cover incidents where an outsider is abusing an insider’s access, which is the case for many persistent attacks, which often happen in smaller stages and rarely all at once.

Another area of concern centers on network traffic. Organizations need to monitor traffic going in and out. Most times, inbound traffic is the focus, but if an attacker gets past that monitoring, there is nothing watching what goes out. Outbound traffic monitoring might help catch exfiltration, which is the difference between dealing with an incident today, or dealing with it next year when your organization is notified by an external party that you’ve been compromised for some time, and you’re still leaking data.

A few of those we spoke with also stressed the need to have a single point of contact internally during an incident, who will coordinate tasks, and ensure that everyone who needs to be kept in the loops has the information they need. If the incident requires notification, then there needs to be a clearly defined message to customers and partners (or the public at large), and even then, only the company spoke’s person (often internal PR or an agency) should speak for the company — no one else. The point of contact, message, and task delegation will change depending on the incident, so organizations should have a few options in place for such an occasion.

Logging is a process that many of those we spoke with for this article had strong feelings about. If you read the various breach reports, it’s often pointed out that evidence of the attack was readily available in the logs, but no one checked them. Worse, in some cases the organization wasn’t even aware the logs were there.

The logs that are most important are Syslogs, but IPS/IDS logs are too, as are the logs on any firewall that has been deployed. These will tell you where the attacker came from, what they did, and in some cases how they did it. Theyre not perfect, but they are a tool that can be used. In addition to those, if your organization uses any type of proxy, the logs from it, as well as VPN, DNS, and router logs, are equally as important. In short, logs from any device, service, or application on the network (including Active Directory) should be monitored and checked.

At the same time, logs can be a nightmare, because organizations need to quickly separate the signal form the noise quickly during an incident. There are a few tools, commercially and open source that are available to help with this process — but not every organization needs them. Sometimes, the organization is small enough that manual filtering and checks are enough. Otherwise, logging and intelligence gathering should be a discussion that is had internally, and if tools are needed they should be vetted and purchased.

At Black Hat this week, HBGary (a subsidiary of ManTech) released a commercial tool for Incident response, which came to be after ManTech Cyber Solutions purchased Vantos. The Seattle-based Vantos had largely used their incident response command center for law enforcement, e-commerce, casinos and resorts.

Vantos platform enables organizations to connect data streams form all of the previously mentioned log points, as well as any other log systems, ticketing systems, and put them in to a simple looking display that can be shared with others who need information, but have next to no clue when it comes to the technical side of incident response.

Passive interviews and conversations with Black Hat attendees also gave us some insight into where they look for information on incident response. A majority of those mentioned SANS, and from that, CSO learned of a useful whitepaper on creating a SIEM and incident response toolkit using open source tools.

While the whitepaper may not be for everyone, its certainly worth a read. In addition to that single whitepaper, SANS as an entire series on incident response, covering issues from Zero-Day threats, social media, indicators of compromise, and cloud-based incidents.

Attendees CSO spoke to also mentioned that professional groups, such as those on LinkedIn, or those within their local community — NAISG (National Information Security Group), are also helpful for gaining insight on incident response, from tools to learning what hasn’t worked in the past.

When it comes to incident response, the bottom line basics are a good starting point. The following were collected via arranged interviews and passive conversations during Black Hat 2013.

• Know your target data: This means understanding all of the types of data on the network, where it lives on the network, and prioritize its value. After that, map all of the ways this data can be accessed both internally and externally. Plan protection for this data, at rest and in motion, and control (as well as monitor) all access to it.
• Document plans for various scenarios: Not every incident will be purely about someone hacking you. Organizations should plan for external attacks, as well as incidents stemming from lost or stolen assets, accidents, and malicious actors from within (including when an outsider compromises and insider’s access.)
• Establish a base of operations: A base of operations is a command center of sorts. This will make things easier for the organization, and often amounts to nothing more than a conference room, or the largest office in the building. It could be the back dock if necessary, but as long as one is established, that’s what’s important.
• Nominate a single point of contact: Make sure this point of contact as access to everyone working the incident, and access to those who need regular updates and information. It’s also wise to make sure there is fast access to PR should it be needed. If PR is needed, from that point on they should be the only voice speaking for the company to the public regarding any incident. Let the legal team deal with issues of compliance and regulatory as needed.
• Update and maintain: Make sure that the incident response plans are updated regularly, and that information is kept current. If new assets are added to the network or new employees added, make sure the plan reflects any relevant changes. This should be done yearly at the least.

Lastly, no matter how good the plan is, it never survives its first real test. Make sure there is an after action report made, and that any mistakes, problems, or failures are learned from. Adjust plans and policies as needed.