The SOC teams responsible for detecting and stopping breaches are famously short-staffed. That\u2019s troubling, because data breaches were up dramatically in 2017.\u00a0With each passing year, the threat detection problem seems to go unsolved, if not getting worse.This is fundamentally caused by the fact that SOC teams have much more data than they can handle or know what to do with. This happens for a few reasons:SOC teams don\u2019t have expertise to build high quality rules\u201cOut of the box\u201d rules don\u2019t work; they end up generating too many false positivesThey don\u2019t have the bandwidth to go threat huntingGartner\u2019s Anton Chuvakin believes that only 0.1 percent of organizations will have the capabilities to be successful at threat hunting on their own.If SOC teams don\u2019t have the manpower to proactively hunt down new threats, what if we could teach machines to hunt? Use human analysts to train and tune the automation, to scale their expertise and get the job done.But what exactly is the job that needs to be done? What are we really automating when we automate threat hunting?Threat hunting is proactively searching for indications of any IT security threat or compromise.\u00a0 It is about filling in the gaps in the SIEM rule set. It\u2019s accounting for the false negatives\u2014the lack of alerts in situations when alerts really should be raised\u2014in rule-based security systems.Threat hunting is successful when SOCs are able to detect the vast majority of threats in their data, in a very timely fashion.Threat hunting is a classification problemIn their sleuthing for threats, SOC teams aren\u2019t starting from scratch. They have lots of data to work with, including log files, SIEM rules and alerts, and other data on the security status of their IT infrastructure.To find new threats, SOC teams need to analyze security events, billions of which occur every day in a large enterprise.To analyze an event, they need to consider all associated attributes or factors. It\u2019s these factors that make an event interesting or not from the IT security point of view.For example, one factor might be that an event occurred at a certain time. Another factor might be that the event involved a certain device. Another factor might be that it involved network traffic from a specific user, and so on. Some factors might be explicit; others might be latent, recognizable only when correlated with other factors.Factors differentiate one event from another. Identifying these factors and labeling them is sometimes known as enriching data. The more factors you have and the richer your data set, the better the quality of your data is for threat hunting.When examining factors, security analysts can\u2019t afford to rely on sampling, which produces errors about 3 percent of the time. Breaches occur in 0.0003 percent of security events, so sampling with an error rate of 3 percent will miss too many indications of possible breaches.Instead of sampling security, threat hunting relies on scoring. Security analysts assign each factor a numerical score that estimates the factor\u2019s importance.For example, let\u2019s say a file is created on a server. The name of the file might be quite conventional, earning that file-name factor a rating of 1. The fact that the file was created by a superuser might earn the file-creator factor a rating of 10. Because the file was created at 3 am on a Saturday morning when the data center was empty, analysts might decide to assign the factor of the time of file creation a rating of 10, as well.It\u2019s through this detailed scoring of factors associated with events that threat hunters can identify anomalous events that bear closer inspection. Scores of factors are combined, and the high-scoring outliers clearly stand out, signaling the possibility of a threat.Challenges with classifying eventsThat\u2019s how factor scoring works in theory. In practice, classifying event data is time-consuming and difficult. There are several reasons why analysts encounter challenges with classifying events.Managing the scope and variety of eventsA single application such as NetSuite or Oracle might have thousands of associated events and each of those events many have many features. Those features might change with each new software release or added feature. The sheer scope and variety of the billions of events that occur each day in a modern enterprise makes classifying security events a daunting task.Identifying relevant factorsNot all factors are equally relevant for the purposes of threat detection. Organizations need to ensure their analysis is focusing on the factors that matter. In practice, this requires a knowledge of the organization itself, its normal operations and the usage patterns of its IT resources.Determining how to enrich dataThe value of enriching data to make it more useful for predictive analytics is important. Knowing which data to combine, correct or extend is critical, but also very difficult. It often requires human intervention and can be time-consuming work.How software automation can helpEvery skilled security analyst whose work includes threat hunting follows a predictable process. They search for high-scoring anomalous events, and if they find any, they examine them closely to determine if they indicate the presence of a genuine threat.Software can apply these same strategies and tactics. You can train software to look for anomalies in events, and to examine them for indications of threats. What\u2019s more, through automation, you can scale up this analysis so that a vast number of events are examined in a fraction of the time required by humans. If the automated threat hunting is based on machine learning, you can train the software to accept and apply corrections and refinements from security analysts, so that the automated threat hunting becomes increasingly fast and accurate over time.Automating threat hunting addresses the three challenges listed above.Automate event analysisAutomated analysis can dramatically broaden the scope of events being examined. Good automation platforms can rapidly analyze millions or billions of events.Automate factor identificationMachine learning can identify factors that bear analysis either through explicit labeling provided by an analyst or through discovery performed by the software itself.Automate data enrichmentSoftware can help with enriching data by automatically clustering similar events together and performing root-cause analysis.New security automation solutions can apply this approach to threat hunting, ranking events by order of severity, raising alerts to any suspicious events that the software has deemed suspicious or an outright threat, and enabling SOC teams to focus immediately on the most critical threats before they can inflict damage.Automating threat hunting does not eliminate the need for human security analysts. On the contrary, human analysts will always be needed to provide the most comprehensive and grounded understanding of context and priority and to make critical decisions in ambiguous circumstances. What automation does do is free analysts to focus on the most critical events more quickly than ever before. It enables confidence that while they focus on specific events, threat hunting is continuously occurring on a vast, automated scale in the background to help keep the enterprise safe.