Avoiding burnout: Ten tips for hackers working incident response

Recent security graduates entering the world of incident response, or those with a strong security background making a career move, face a challenging environment that often leads to frustration and burnout.

Speaking from personal experience, Lesley Carhart, the Incident Response team lead for Motorola Solutions’ Security Operations Center (SOC), will address the burnout associated with incident response careers, by putting special focus on those entering the profession for the first time, during a presentation at CircleCityCon in Indianapolis this June.

Carhart’s talk looks at the human side of incident response, and avoids discussion of methodology. Instead, she’s created a roadmap based on “lessons learned” from her time in the trenches, and how these lessons can be used to build (not burn) bridges between the security and business worlds.

In an outline of her talk for CSO Online, Carhart said that she has seen numerous examples of burnout within the security community and in the SOC as they train newcomers.

“While effort is being made at a regulatory level to provide organizational incident response processes and procedures, little attention is being paid to training human beings to be better equipped incident responders,” she said.

Moreover, while there is plenty of public knowledge and training available when it comes to incident response methodology, none of that fully prepares newcomers to the field; especially when it comes to dealing with complexity, stress, or the relationship between security and the business.

“By not providing adequate foundation for dealing with these problems, we’re failing the next generation of DFIR professionals. Bright young people who have excellent technical education often fail or quit when they are faced with corporate bureaucracy, high pressure cases, and a lack of business skills,” she explained.

Nobody Cares About My Exploit

“I’m going to start with the most contentious of my rules, yet the one that I personally find causes the most stress and failure to technical people moving into incident response,” Carhart says.

The story starts with a brilliant young hacker, a new hire that has developed a new tool, finds an interesting vulnerability, or develops a plan for dealing with some variant of a given piece of malware. When these findings, tools, or plans are presented to management, the hacker is told to get back to work.

“It would take a sociologist to say whether the average hacker personality type is partially a byproduct of the millennial generation, or something totally unique. Regardless, security professionals tend to be creative, intelligent, and independent,” she added.

But these traits don’t always mesh with corporate society, so the first rule that new incident responders need to learn is how to sell things to their organizations. While accomplishments matter, they have to be presented in a way that quantifies value.

“We have to understand, at the most basic level, what our organizations care about. At very simplest, almost all want to make money and avoid losing money. In rare and specific cases, they might care about life and death. So, we must quantify things in these terms when we present it to a company, whether it is the number of dollars our tool could earn, the hours of work that will be lost if preventative actions aren’t taken, or the number of lives that will be saved if our military operation succeeds,” Carhart explains.

Build Good Relationships

One harsh reality is that security doesn’t win every battle in business.

However, while most security departments exist within a vacuum to a degree, the incident response program can’t. Incident responders and handlers need to coordinate their efforts, from IT to legal, and to engineering and marketing. Moreover, incident response programs need support from these areas.

“Even when the people we are relying on are not technically skilled or resist remediation steps, it is crucial that we remain calm and courteous,” Carhart says.

“While the first thought under stress can often be to lash out at nonresponsive teams, it is very likely in most incident response roles that we will interface with the same people repeatedly. We should never burn bridges. On numerous occasions, I’ve had to go back to obscure IT staff who were involved in a previous incident for assistance on another case. They’re much more likely to do favors for me if I have been polite and helpful in the past.”

Another tip is to make good connections within your local security community, as well as the online community. This opens up several avenues of support including correlation of malware samples, potential training opportunities, and information on new vulnerabilities or exploits.

Communication Matters

While keeping your cool and remaining polite, remember that verbal and written communication is not only a required skill within incident response, it’s also a major weakness.

“Computer security is a specialized and demanding field which often attracts people who aren’t particularly interested in language studies. Unfortunately, upon the move to incident response, not only must we deal with many different teams, but we must also deal with all levels of technical expertise. In fact, many of the teams I often rely on are non-technical: lawyers, public relations, and even human resources,” Carhart says.

You don’t have to be an English major, but you do need to have the ability to express yourself in a way that non-technical people can fully understand. People will judge you based on the way you communicate.

“This means an attempt at proper spelling, grammar, and punctuation in written communication and in the countless reports which are required of an incident responder. The bottom line is that people will go back to the people they can understand for help, with praise, and with opportunities for advancement.”

Assume Nothing!

“Not only is an incident responder presented information from end users, but from numerous other teams, from technical to non-technical. Even the smartest malware analysts, engineers, forensic analysts, and C-level executives can and will make mistakes. During a stressful, hectic security incident, it can be incredibly tempting to take facts or conclusions we’re presented with at face value instead of verifying or considering them,” Carhart says.

The scientific method should be applied to incident response. Start from the beginning, gather and verify all of the facts, and then confirm or correct anything that doesn’t seem right.

“One incorrect assumption can cause our entire investigation to go in the wrong direction, and totally derail remediation. As security professionals, we’re constantly bombarded with horror stories and paranoia. This also can easily influence the conclusions we draw early on in an investigation.”

In short, don’t find facts to fit a conclusion, draw a conclusion from the facts.

Other assumptions to avoid include the line of thought that everyone has the necessary skills and experience to do technical tasks incident response teams consider simple. Moreover, there’s the assumption that everyone has the tools and resources necessary to perform those tasks. Both are far removed from the truth.

Educate and Document

Incident response teams need to keep rigorous notes on the training, day-to-day work, and incidents. This includes any task the team does more than once, tasks only one person knows how to do, and lessons learned after every incident.

The point, Carhart said, is that being able to quickly reference processes and notes will save us added stress and confusion while working under pressure. Moreover, management should allow time and funding to train their security team. But there’s more to it than simply attending a class or taking a certification test.

“Incident response skills should be drilled until they are easily recalled under pressure,” she said.

“Police and military train with firearms until shooting properly is second nature. Disaster response personnel have regular response and mass casualty training exercises. Firefighters practice in condemned buildings. The objective is to do critical tasks and processes so many times that in a high stress environment, they can be done without much effort. There is no reason we should not be doing the same.”

However, education for end users and other business units is just as important. The more support and comprehension that an incident response team receives from the rest of the organization, the easier their jobs will be.

Further, incident responders need to take a personal responsibility in their own training and security knowledge. To put it bluntly, there is no excuse for an incident responder that lacks a general understanding about what’s making headlines that day in security news. Access to such information is just too readily available.

Triage Intelligently

Often overlooked in training, triage is perhaps the most basic and integral skill an incident response team can have.

“There are overwhelming days as an incident responder when everything goes wrong simultaneously. Fortunately, the skills used in triage in other industries are fairly universal and many can easily be applied to security,” Carhart said.

“First of all, the few minutes taken during an incident to breathe and figure out next steps are almost always worth it. It’s at this point, during triage, that our previously documented processes, methodology, and training become key.”

The recommendation is to use a risk matrix, which will offer a rough evaluation of an incident’s risk and priority ranging from low to extreme. This triage method also enables the incident response teams to explain their decisions to management.

Don’t Panic!

To get a better understanding of this item, one should look at the two types of people who panic during a security incident.

The first are the extremely technical people who have found minutiae. They may be experts in their fields, but they’re likely missing the big picture, or failed to stop and think about the situation.

The second group are the non-technical people who have seen something that’s alarmed them. This could have come from an email, a news article, or a conversation in passing.

“We as security people can fall into both groups depending on the situation – no person is an expert at every field. Panicked security (as with anything else), is not good security. Once again, we fall back to our scientific method, and ‘peer review’ any information we are receiving that doesn’t seem factual or reliable,” Carhart says.

“This is a good time to bring up the distinction between ‘Incident Responders’ and ‘Incident Handlers’, again. In an ideal situation, both job functions should exist, and the Handler should be talking to panicking team members, management, and end users while the Responder works on the investigation relatively uninterrupted.”

Remember the basics: Practice, good documentation and training, and triage skills. When feeling overwhelmed, incident response teams should fall back on these lifelines.

Specialization is for Insects

“The trend in information security is to start out as a log analyst, and then move to a specialized role such as penetration testing, malware analysis, exploit development, or digital forensics. Most people are drawn more to some of these fields than others. As an incident responder, we must return to being a generalist. This isn’t to say that specialized expertise isn’t useful. However, the risk is that our specialization will color the way we perceive an incident,” Carhart explained.

“My expertise is in digital forensics. If I look at the same compromised system as a forensic analyst as my colleague who is a malware analyst, I may see a very different case. While he will see the details of malware binaries and their communication on the compromised system, I will note files modified, deleted, and moved, and remote systems accessed.”

To be effective, both must see the incident through the eyes of a malware analyst, network analyst, penetration tester, and forensic analyst. At the same time, neither has to become an expert at these skills, there are resources available to help when needed. But a generalist level of knowledge is important, if only to understand what steps need to be taken next.

See the Big Picture

There are many different types of security incidents, Carhart explains, everything from targeted attacks, to malware outbreaks and insider threats.

Each one will have a different business impact on different parts of a victim organization, and an incident responder has a responsibility to understand the big business picture.

“He or she should be generally cognizant of systems and projects that may be targets or of interest to an attacker, and able to know who to contact for further information about them. The incident responder should also be doing enough due diligence to understand the cultural, legal, environmental, and financial differences between our business units and physical locations. This will allow the security team to understand how incidents will impact each one differently and how they will need to be remediated differently,” she said.

Keep Control of the Incident

An incident responder leads the coordination of response and remediation, and ensures that the various teams responsible stay on track and on schedule. They’re the ones who determine where the investigation leads next.

The incident handler’s responsibility will be to schedule regular communication and status updates with all impacted teams. This aspect of the job can require some project and people management skills that many universities don’t offer to information security students.

“I’d suggest to anyone struggling with dealing with task scheduling or dealing with senior leadership to take a basic management course. Attending security conferences and meet-ups (especially speaking at conferences), can also help build these skills,” Carhart explained.

“Lastly, an incident responder should take pride in his or her work, and responsibility for his or her mistakes. Every incident is different, and every person will eventually make mistakes. The best thing we can do is learn from them and not repeat them.”