Analytics and Forensics: The New Power Couple for Threat Hunting

 A SANS survey reveals that of the 86% of organizations that are threat hunting in some form or the other, a majority – 53% – are doing so in an ad hoc fashion. While this is one way for security teams to build expertise, it is not an effective long-term strategy – that needs a formal proactive threat hunting program based on well-defined methodologies. Proactive threat hunting, which involves hypothesizing and analyzing data and testing to find threats, is more effective at securing organizations than reactive threat hunting, which is a response to a flagged threat. That then begs the question: why don’t more organizations have proactive threat hunting programs?

The reason is threat hunting is incredibly challenging at organizations that rely on traditional security products.

But first, some background. Cybercriminals employ constantly evolving techniques in slowly gestating multi-stage attacks to subvert an organization’s defenses but, however good they are at subterfuge, they leave clues all across the organization. Threat hunters ferret out these clues and piece them together to reveal the attack. They are a rare breed, drawn to this work because it is challenging and requires creative thinking. Threat hunters are the chief detectives of the security world today. The continuously changing and interesting detective work keeps them engaged in their jobs.

Unfortunately having to deal with mundane tasks is the most common obstacle faced by threat hunters at organizations that are using traditional security products. Before they can get to the really fascinating work of threat hunting, they must first suffer through cumbersome and repetitive tasks. Stuff like finding where across the organizations all the necessary information lives, consolidating it from a variety of siloed security systems, ensuring it meets the limits of comprehensiveness that is artificially imposed by legacy systems (while knowing that that truly comprehensive information would make threat hunting more effective and fun), and attempting to extract meaningful insights from these limited data sets.

This hurdle could easily be rendered moot if organizations took a modern approach to security, one that intrinsically ties together advanced analytics and long-term full fidelity network traffic. Many vendors are touting advanced analytics in their products but without the underpinnings of long-term and full fidelity forensics there are three notable benefits that aren’t achievable.

State Machine-based Analytics Easily Reveal Multi-Stage Attacks

First, retention of long term forensics enables analytics to incorporate the benefits of a state machine in results. This is imperative for detecting advanced attacks, which involve multiple stages that typically occur over extended time periods. These multi-stage attacks, also known as advanced persistent threats (APTs), represent over a third of all attacks according to a recent report, so it has to be an area of focus for security teams. In such an attack, cybercriminals would execute a variety of actions, which individually may appear innocuous but when evaluated as a complete sequence would reveal the attack. Analytics combined with long term forensics and a state machine can keep track of multiple sequence of events and determine whether an alarm is a one-off anomaly or part of a larger attack.

In contrast, analytic products, including those that use advanced techniques, that cannot maintain long-term state would not be able to present analysts with a picture of the entire attack. At most, they’d generate alarms for all individual actions. For short staffed security teams, who are already suffering from alert fatigue, additional warning bells that require significant investigative efforts to resolve would simply slip into the noise and result in the attack being missed.

Full Fidelity Forensics Enable Better Visibility

Next, is the fidelity of forensics which enables better visibility into threat activity. There is an incredible range of metadata attributes (e.g., HTTP referrer, email sender, IP source, date and time stamps, etc.) embedded in network traffic. Using analytics on this spectrum of attributes better positions analysts to thwart attackers. For example, monitoring SMTP traffic and payloads to build a baseline of normal user behavior and then identifying deviations that can be used in concert with other analytics to surface security events or serve as data points that threat hunters can use in hypothesis testing.

This is an opportunity to highlight another point: the goal of analytics shouldn’t only be about identifying threats. It should also be about filling in the missing information that is hindering threat hunters from realizing their full potential. Making this higher order information – like the hosts that started a flow with one protocol and then changed to something else mid-stream, hosts connecting to port 443 and not using the SSL protocol, and so on – easily available to threat hunters is invaluable for unearthing actions of sophisticated attackers.

Long Term Forensics Make Retrospective Analysis Possible

The median breach detection window is 146 days yet most organizations using legacy products don’t retain forensics for anywhere close to that length of time. There are good reasons for this: legacy products are difficult to deploy, configure and support, and impose prohibitive costs to retain forensics for the necessary length of time. However, it doesn’t change the fact that these organizations can’t figure out the compromised assets or even if they’ve been affected.

A modern approach to security makes retention of forensics for time periods longer than the breach detection window possible, which is a tremendous boon for threat hunters. Suddenly they are working with a full deck of cards. If news of a new vulnerability becomes available they can search for any occurrence of that in their historical records and confidently determine if there has been any impact. Or they can build their own hypotheses and test them to discover previously unknown threats, catching them early in the Cyber Kill chain to mitigate their impact.