• United States



by Simone Kaplan

Incident Response: When Bad Things Happen to Good Companies

May 01, 200312 mins
Network Security

If you don't have a clear incident response plan in place, you risk losing millions of dollars.

Were there a computer Incident Hall of Fame, you could probably imagine strolling the halls and browsing through exhibits of history’s most dynamic electronic viruses and worms—the villains whose names have sent shivers down the spine of any security expert equipped with a decent memory: The Morris Worm, Melissa, Nimda, Code Red, LoveLetter, Klez and, of course, the most recent inductee, SQL Slammer. You might also see some of the more notorious service outages, hacker penetrations, denials of service, malicious e-mail and Internet attacks on display. All have caused varying degrees of chaos, and some have even stopped businesses in their tracks, crippling productivity and costing millions of dollars in lost commerce.

And yet all could have been tamed. Had someone the foresight to put an incident response plan in place, those viruses and worms and outages and attacks might not be so infamous today.

Of course, such a place doesn’t really exist, but the threat of cyberattacks does. And it’s growing every day, due in part to the widespread use of e-mail and the Internet. According to statistics from Carnegie Mellon’s CERT Coordination Center (CERT/CC), the number of reported cyberincidents has surged from only six in 1988 to a whopping 82,000 in 2002. Despite the rising threat, however, CERT/CC finds that most CSOs don’t even think about their response to an incident until after they’ve experienced an intrusion of some sort, says Chad Dougherty, an Internet security analyst at CERT/CC. “That’s because most companies feel relatively safe. They believe that the hackers won’t target them, specifically,” he says.

But they’d be wrong, says Dougherty. The majority of computer incidents are no longer focused on a particular company. “Most attacks now are automated,” he says. “They spread with the intent to damage everyone and everything they can.”

Clearly, it’s time for CSOs to come to terms with the need for response planning. “For a long time, incident response meant having a loose team of people on call if something went wrong,” says Gene Fredriksen, vice president of information security at Raymond James Financial. “Then companies started getting hit regularly, and I think CSOs are finally beginning to realize that incident response is not optional.”

Not optional, but also not easy. Even a well-prepared CSO knows that an incident response plan can’t keep his company completely safe from attackeven with the latest tools for intrusion detection. “There’s just no such thing as zero risk,” says Leslie Macartney, CISO for Reuters. “And you can’t always predict the number, nature or severity of the attacks. But incident response plans are necessary because, in short, no matter how much you try, things will occasionally go wrong. Your company is at its greatest exposure in the time between when an incident occurs and when the containment actions are completedthat’s when most of the damage occurs.”

And it’s not just an internal matter, says Macartney. “Customer confidence can be damaged if it appears the company has been remiss in its handling of security events. The company’s reputation could be at stake.”

But you can’t protect everything completely, so you must prioritize, Macartney adds. By creating a specific strategy that states what to prioritize and how to react if an incident does happenand by making your security organization capable of detecting, analyzing, and responding quickly and knowledgeably to an eventyou can limit the damage done and lower the costs of recovery. And then, by knowing who to call and what to do next, you can decrease the amount of time it takes to recover and possibly save you and your staff from additional disasters along the way.

“The organizations that don’t know how to respond to incidents are the ones that will really get hurt,” says Kevin Connell, director of information security for the shared data center of the Securities Industry Automation Corp., which runs the computer systems and communications networks of the New York and American stock exchanges. “And while it’s hard to protect against something you can’t predict, it’s not so hard to react decisively in crisis situations once you have a plan in place and a procedure to follow.”Getting StartedWhen thinking about incident response planning, remember that the best defense is a good offense. But before you do anything, says Ariel Silverstone, CISO at Temple University, it’s important to define the nature of a cyberattack. That way, you can decide what constitutes an incident for your company (see “What’s It to You?” at Generally speaking, a computer incident is anything that potentially compromises the confidentiality, integrity or availability of a computer system. Sometimes such incidents can be reallike a service outage. Other times, the incident is merely a perceived attacklike when a file disappears because an employee simply moved it from one server to another without telling anyone.

Drafting the response plan includes four main activities, according to Kenneth van Wyk, coauthor of Incident Response and director of technology for Tekmark Global Service’s technology risk management practice. First, pull together a response team that broadly represents the entire organizationHR, legal, media relationsand build a phone list to make alerting the necessary people more efficient. Then, create an incident reporting forma checklist of sortsto help document the incident and track costs along the way. Next, build a flow chart detailing the process that the team should follow during an incident (see chart, Page 56). And finally, map out a post-incident review process to ensure continuous improvement with your overall plan. Each part will play an important role in helping you deal with incidents before, during and after they occur.Go TeamIncident response teams go by different names in different companies: Some call it the IRT; others use the acronym CIRT or CSIRT, for computer security incident response team. Whatever you call it, the group is pretty much your version of a SWAT team, called into action when a computer incident occurs.

Because every incident (and its potential effect on your systems) has its own particular traits and required responses, it’s important to first get a grasp of the kind of incident-handling expertise your network staff and others on the team already have, says Walt Foultz, director of IT security for Farmers Insurance Group. “Incident response is not only a security activity,” he says. “All sources of qualified and competent assistance must be assessed so you can be sure, collectively, that you have the skills to handle the job.”

During the early stages of creating an incident response program, Foultz suggests surveying your potential team members to scope out the depth of their incident response skills and technical knowledge. Find out if anyone has a specialty, such as dealing with network probes or e-mail viruses. Foultz gives his own staff verbal pop quizzes to make sure they know their stuff. “One technique I use is to set up hypothetical situations, and they have to tell me what they’d do,” he says. He also makes sure every staff member allocates a percentage of her regular work time to learn about the latest cyberincident trends and security technologies. “We do that with individual training and by disseminating internal research to the team through management and scheduled awareness sessions,” he says.

How your team is structured depends on the skills and available resources within your company. Large companies often have response teams staffed with people dedicated solely to handling incidents, while smaller companies often create a team consisting of a core group of people from several IT and business departments who get tapped if something happens.

George Wade, Lucent Technologies’ regional security manager for North America, recommends casting a wide net when choosing your incident response team. The ideal team should include members of your IT security team who know the company’s networks, applications and systems inside and out. Don’t forget to include representatives from other departments in the company. Not all CSOs will include people from media relations on their response teams, Wade says. “But if someone defaces your corporate website and reporters suddenly start calling, you’ll understand very quickly how important it is to have a company spokesperson informed and involved,” he explains.

Some companies decide to involve their disaster recovery or business continuity departments in their response teamsthe reason is that other voices often prove helpful when things really go wrong and systems need to be shut down completely.

The team also needs a certain degree of flexibility. “Response teams shouldn’t be static,” Wade says. “They can be added to or subtracted from at any time if you decide that something needs to change.”

Once the team is in place, you’ll need to create a contact lista staple of any response plan, says van Wyk. “If you overlook creating one, you do so at your peril,” he says. It’s essentially a phone tree, including emergency phone, pager and e-mail information for members of your incident response team. The list should also include contact information for outside authorities, such as local and state police, the FBI, CERT and any third-party provider that your company may rely on for backup assistance. Contacting the authorities won’t be necessary for every incident, but it’s good to have the information at your fingertips.

For continuity purposes, list contacts according to job function, authority and skill set rather than by name. That way, if someone leaves the company, you won’t have to rework the entire list. It also means that there’s a clear reporting structure in place: When an incident occurs at 3 a.m., for instance, and the system administrator sleeps through his pager alarm, the team member who discovers the incident can quickly alert the next person in the chain of command.Go with the FlowOnce your team is in place, you should create a diagram that spells out, step-by-step, what each part of the security organization needs to do when a breach occurs. And while the incident process needs to be flexible in order to handle various kinds of attacks, Silverstone says, you won’t want to leave any of the steps in the diagram to interpretation. “Be precise. Everyone should know who to call and what to do in every type of situation,” Silverstone says. “If you leave it open-ended and someone makes the wrong decision, you’ll leave your organization open to liability.”

Once you determine that you have a genuine incident on your hands, you and your designated team members can formulate a response strategy. Is the incident major or minor? Does it threaten vital business functions? Do you want to contain the incident and maintain business continuity, or do you want to allow the incident to unfold so that you can gather forensic evidence for an investigation further down the road? Should you contact outside agencies yet? Is it necessary to communicate with the general employee population? The answers to such questions will help the process move along more quickly and predictably, saving precious time and money, minimizing damage and maintaining business continuity for your company.

Consider making a team member the designated note-taker so that when a crisis hits, there’s no confusion about who’s capturing all the important information.It Ain’t Over ‘Til It’s OverFinally, after every incident, CSOs need to lead their incident response teams in a postmortem review process that examines how well the incident team dealt with the attack. Did team members follow the response diagram? Did staff members handle the incident calmly? Did everyone on the contact list respond promptly? Should the contact information be updated or changed in any way? And, finally, do you need to add anyone to the team or adjust the procedures?

“If you don’t learn from what you’ve just experienced, you open yourself up to more attacks,” says Raymond James Financial’s Fredriksen. The review is your chance to improve the plan and the team so that you can work out any kinks before the next incident strikes. Fredriksen recommends doing a risk analysis after every incident to make sure as many vulnerabilities as possible are secured.

After the review, you will find it useful to complete an incident report for your records. Among other details, the report should include all the information you’ve gathered about the incident, both during the response process and in the postmortem. That way, if you decide to pursue an investigation, you’ll have all the evidence on hand.

Remember that the steps to a clear, planned response are not complicated. Once you are sure that an incident has actually happened, determine whether it’s a major or minor event.

Decide whether your priority is to pursue an investigation and allow the incident to play out, or to shut down the problem as quickly as possible.

And finally, work to defend against further attacks. Take a look at the way in which the attack happened and determine if an application needs to be patched or a port reconfigured. Take whatever action is necessary to prevent the attack from happening again. And be sure to let everyone on the response team know that the problem is fixed.

IT threats may be coming faster and faster. But by having a clearly defined response process, you can prevent attacks from devastating your systems. “Plans are not a panacea,” Reuters’ Macartney says. “But if you use them strategically, you can limit your exposure to risk.”