Who wants to go threat hunting?

I’ve been a lot of things in my professional career including paramedic, accountant, computer trainer, PC/network technician, VP of IT, consultant and writer. The most enjoyable job I ever had was penetration tester. You get paid to break into places, work with cool people, and learn a lot. Best of all, if you couldn’t break into a place, the customer would be delighted and brag about how their computer security defenses didn’t fall to a sustained hacking test. 

Over the last few years, an even more elite group of whitehat hackers have emerged: the threat hunters. Threat hunters are part proactive hacker, part forensic investigator, part intrusion detector, and part incident responder (IR), with emphasis on the last role.

I interviewed Rob Lee about what it takes to be a threat hunter. Lee is a Boston-area author, consultant, and SANS Faculty Fellow with more than 18 years of experience in digital forensics, vulnerability and exploit discovery, intrusion detection/prevention, and incident response. He began his career chasing bad guys in the Air Force Office of Special Investigations, the U.S. Department of Defense, and was a director for Mandiant, focusing on advanced persistent threats (APTs).

Lee co-authored the second edition of Know Your Enemy, one of my favorite honeypot books, and he co-authored the Mandiant threat intelligence report, M-Trends: The Advanced Persistent Threat. He’s also one of the original threat hunters, before threat hunting was a term.

When did you become a threat hunter?

Lee: As part of the response team to what became known as the Moonlight Maze Russian hacking offensive in 1998. It was the first time a known nation-state actor, in this case Russia, started to actively compromise multiple government sites for nation-state reasons. Before this, most hacking was defacing websites and things like that, but these were real professional adversaries. They didn’t run.

Rob Lee

Back in the day, when you caught hackers, they used to get out of the system or network and never come back. These hackers, the Russians, didn’t run. They did go silent, for a month or two, but they never left. Today this is the norm, but at the time it was a brand new [reaction]. They learned what we were doing, watching us, even going so far as to keystroke log what we did, and then started up their activities again. We learned from watching them. They were extremely patient. They watched us. We watched them. We learned a lot about nation-state adversary groups. We had to change how we did incident response.

What is a threat hunter?

Lee: A threat hunter is a proactive incident responder. An incident responder waits until they get notified of an incident to get involved. A threat hunter hunts for bad guys before you know there is an incident. They have some information about the bad guy, where they are likely to attack and how, and then go looking for them. Threat hunters are incident responders and forensic investigators actively looking for new threats before traditional intrusion detection methods can find them.

Threat hunters are an early warning system. They shorten the threat’s “dwell time,” which is the time from the initial breach until they are detected. In the past, threats were often inside of networks for months without detection. [RG: often years] A threat hunting team is trying to shorten the time to discovery.

How long has the term threat hunter been around? Is it an actual job title?

Lee: It’s been around six years. Working at Mandiant around all the nation-state attacks led to proactive threat hunting. We were proactively looking for adversaries. We’re going hunting for bad guys. It’s become a more popular buzzword over the last three years. I see it everywhere, including in job listings and at security conferences, including SANS’ own Threat Hunting and Incident Response summits. 

How does someone become a threat hunter?

Lee: To become a threat hunter, one must first work as a security analyst and likely graduate into IR and cyber threat intelligence fields. Combined with a bit of knowledge of attacker methodology and tactics, threat hunting becomes a very coveted skill. Threat hunting is one of the most advanced skillsets one could obtain in information security today. The core skills of a threat hunter include security operations and analytics, IR and remediation, attacker methodology, and cyber threat intelligence capabilities. Combined, a hunter is the special operations team of an organization’s defensive and detection capabilities.

A threat hunter is taking the traditional indicators of compromise (IoC) and instead of passively waiting to detect them, is aggressively going out looking for them. Traditional intrusion detection doesn’t do a great job on the crafty adversary. They will avoid tripping the normal intrusion detection defenses. It takes a threat hunter to find them.

Should every company have a threat hunter?

Lee: Not every company can have one. It takes a certain size and sophistication. First, you need a real, dedicated security operations center (SOC) — not a little or part-time one, but one that can create, consume, and utilize threat intelligence, and that understands the likely adversary, not just indicators of compromise. Threat hunting teams need threat intelligence plus a network person, an endpoint person, a malware analyzer, and a scalable bunch of tools. A threat hunting team is like special operation forces.

A threat hunter has to know where to look and what to look for first. It takes a real, dedicated SOC to get that information. If you’re hunting for ducks, you need to know where the ducks are likely to be and what to look for. The same with nation states. If the Russians or Chinese are spying on us, where are they likely to be? What are they looking for? What are their indicators of compromise? Then you send out the special operations force to find them.

How many threat hunters are there?

Lee: It’s hard to assess. Lots of people call themselves that, but for sure the large companies with large-scale security operations do [have them]. A lot of the Fortune 200 companies have them, along with DoD and other government agencies. Most of these entities have multiple threat hunting teams. Many of the big companies like Target and Microsoft, who were victims of large attacks, now have multiple, elite threat hunting teams, each focusing on different business units. They went from not having the best security to having multiple proactive special operation teams. It’s pretty great to see the turnaround. They have some of the world’s best threat hunting people.

Anything else to share about threat hunters?

Lee: Both the companies that employ them and the people that work on threat hunting teams need to understand that they are likely to fail. They are performing a very tough role, looking for a crafty adversary that does not want to be discovered. They might not find the bad guy in time. It’s a very needed role, but quantifying success in a traditional sense can be harder. If they are proactively looking for threats they are doing their job.

More on cyber attacks: