Threat hunting explained: Taking an active approach to defense

Selim Aissi has gotten more aggressive.

Aissi, CISO at software company Ellie Mae, has his security team searching for intruders trying to sneak into his enterprise systems and working to root out any that might be hiding.

He wants to shut attackers down before they have a chance to act.

Selim Aissi

“It’s about making prevention more productive,” he says. “It’s not passive, like detection. We’re proactively searching for these threats so we can isolate them. That’s what makes threat hunting a truly active defense.”

Aissi is one of a growing number of CISOs who have added threat hunting to their cybersecurity programs, seeing this approach as a way to identify intruders who have either eluded monitors or are so new that they’ve yet to set them off.

Threat hunting definition

Threat hunting is the practice of proactively searching for threats that are hiding in an organization’s systems. Experts say threat hunting is becoming an essential element of enterprise security that builds on conventional perimeter defenses that, while still important, are far from foolproof.

Gregory J. Touhill

“The perimeter defense model of security leaves organizations exposed to attacks from the outside and the inside, so CISOs have to make the assumption that they have an attacker in their midst, so you want to be proactive and look for signs of those who are lurking in the shadows,” says Gregory J. Touhill, adjunct faculty member at Carnegie Mellon University’s Heinz College of Information Systems and Public Policy and retired U.S. Air Force brigadier general who served as the first federal government CISO during the Obama administration.

Certainly, security teams have always been tasked with identifying threats to the enterprise and shutting down those that have made their way past organizational defenses.

But security teams historically identify threats once they’ve made themselves known in some way, such as when they’ve tripped an alert from a monitoring system within the security operations center (SOC).  “The average time in system before detection is more than 100 days, so there’s a need to look for that activity,” says Mike Ortlieb, a director of security and privacy at management consulting firm Protiviti. “But threat hunting allows you to thwart the attackers before they succeed.”

Mike Ortlieb

That’s why Ortlieb and other experts highlight the proactive nature of threat hunting, stressing that it helps CISOs and their staff better protect the enterprise.

“Threat hunting is basically a discipline where you’re proactively searching for threats that are hidden, and that’s really important because a lot of nation-state actors and criminal groups and other malicious actors now have the capability of parking on your network and lurking in the dark corners where they can harvest information and study you and your people and your tactics and techniques and obtain data and log-in information. They can sit there and understand your business processes and credentials undetected and then use that information at a time of their choosing to launch an attack. We’ve seen that in both the private and public sectors,” Touhill says.

What’s driving threat hunting now

The goal of the security team has, of course, always been to stop bad things from happening as early as possible, whether that has meant shutting down an attempted hack from the outside or thwarting risky employee behavior.

But CISOs generally acknowledge that their long-practiced strategies have not been as effective as they’d like in stopping attacks.

Consider this startling finding from Arkose Labs, a cybersecurity platform provider. According to its Q2 2020 Fraud and Abuse Report, there were 445 million attacks detected globally in the first quarter of the year, for a 44% increase in total number of attacks.

Such statistics aren’t an anomaly, as studies consistently show cyberattacks are on the rise. At the same time, the enterprise IT stack has become more complex and porous, with the perimeter separating an organization’s systems from the rest of the technology world rapidly disappearing.

Enterprise security teams often struggle to keep up, says Wolfgang Goerlich, advisory CISO for Duo Security, a Cisco business unit, which has offered workshops on threat hunting. SOCs are inundated with alerts about possible problems — so much so that they can’t possibly investigate each and every one. Cisco’s 2020 CISO Benchmark Report, in fact, found that 41% of organizations get more than 10,000 alerts a day.

Wolfgang Goerlich

Alert fatigue sets in and can keep security teams from being as effective as they could be. “If you’re constantly getting pinged, you can never think deeply and you can never think broadly,” Goerlich says.

He also points out that alerts generally indicate active attempts to attack and are not necessarily effective in finding threats that are either waiting for an opportune time to attack or are new and thus unknown to the monitoring systems.

Goerlich says he has seen how an overload of alerts coupled with a strictly reactive approach can leave an organization exposed. He led a red team simulating attacks on a company to test its security posture, using various tactics to try to get into the company’s systems. The security team did indeed identify the individual pieces of the attack, with monitoring systems alerting the SOC to phishing emails and malware. But while the security team successfully stopped individual attempts from exploding into full-blown events, they failed to see the big picture that there was an ongoing, multi-pronged coordinated attack.

“When you’re closing tickets in a fast manner — as you should be doing — you miss the full scale of what’s happening,” Goerlich explains.

But threat hunting, with its proactive approach and its focus across the IT stack versus alerts, helps security teams spot such activity.

Who threat hunting is for

Threat hunting is not new, but it is becoming a staple layer in enterprise security as new technologies such as machine learning have become more mature and mainstream and as threat intelligence has become more prolific and widely available.

“Threat hunting has been democratized, so in the past couple of years more organizations have been looking at it,” Goerlich says.

Cisco’s 2020 CISO Benchmark Report found that 76% of large enterprises have a threat hunting team in place. Another Cisco report, Big Security in a Small Business World: 10 myth busters for SMB cybersecurity, found that 72% of small and midsize organizations have some employees dedicated to threat hunting.

Goerlich says few organizations have a fully mature program, as threat hunting is still an emerging security practice for most.

Furthermore, experts acknowledge that larger organizations generally have more mature threat hunting practices as they’re better able to afford the technology and talent needed for an effective program. Larger organizations are also more likely to have full-time threat hunting operations, whereas smaller enterprises generally run threat hunting as a part-time operation.

The ROI of threat hunting

Goerlich says threat hunting gives security leaders a better understanding of their systems, their vulnerabilities and the hackers’ likely targets — all of which can inform how to implement improvements and better controls.

“The ROI of threat hunting is identifying broader, ongoing attacks and stopping those things that would get past monitoring,” he says. “Threat hunting is deep work; it’s focused work. It’s having a team that is exploring the data to look for patterns of behavior or activities that regular monitoring will miss.”

Studies show threat hunting is effective. The SANS Threat Hunting Survey: The Differing Needs of New and Experience Hunters, released in October 2019, found that 61% of the 575 respondents engaged in threat hunting reported a measurable improvement in their overall security posture.

“Some threat hunting is better than no threat hunting,” Ortlieb says. “You can start out spending some time doing some rudimentary hunts for some things that look abnormal and even that accomplishes two things: you learn more about your own normal activities and you’re also able to possibly find malicious activities.”

People, processes and tools for effective threat hunting

As it is with other pieces of enterprise security, a solid threat hunting program involves a combination of the right people, processes and technology — as well as a commitment to doing the work.

“A good threat hunting team has a mandate to do analytics and investigations that are free from alerts,” Goerlich says. “They have data available from the monitoring team and available tools so they can operate from threat intelligence to understand the criminals, to understand the infrastructure, to understand patterns of behavior and to feed back [their findings] to the incident response function.”

Aissi says he uses various technologies to run his threat hunting operation and leverages tactical, operational and strategic threat intelligence to understand the tactics, techniques and procedures (TTPs) typically used by threat actors. He notes that analytics, intelligent systems and automation are key to building an optimized program.

Key elements to an effective threat hunting program include good threat intelligence and security analytics tools enhanced with machine learning as well as a robust security information and event management (SIEM) practice and forensic capabilities to provide support and to take action on any threats identifying during hunting exercises. These pieces, along with skilled analysts, enable a threat hunting team to identify anomalous activities that have been concealed by bad actors and that often closely mimic legitimate traffic within the enterprise systems.

Where threat hunting sits in a security program

Aissi and others also stress that investments in threat hunting should not come at the expense of other security programs; threat hunting is an addition to conventional defensive practices and other security layers.

“Threat hunting is just one aspect of what you’re doing as part of an overall security plan,” says Theresa Lanowitz, head of evangelism and communications at AT&T Cybersecurity. “Organizations that are implementing really good security practices and tools are taking the necessary first steps. You want to take that layered security approach, which can be very effective in stopping the majority of attacks. But then you get to some advanced threats — there are ones that can get past your other layers — you want to stop those attacks before they happen. Threat hunting delivers that capability.”           

“[Bad actors] are going go quiet and they’re just going to be collecting information. They have their ears on, like submariners, and they just sit on the bottom and they don’t transmit, they’ll just receive. But every once in a while they’ll pack down information and send it out in batch to a command and control server; it will be like a doppelganger, looking like a legitimate site, and if you’re not looking for that you may never see it,” Touhill says. “But if you’re looking for it, you stand a better chance of finding it.”