How Target evolved its threat hunting program: 3 key steps

Threat hunting – proactively searching through your own company’s networks to hunt for attacks that might evade other security measures – often signifies a company with a mature and well-resourced security organization. But just as threat actors are constantly evolving, organizations should be willing to reassess and change their security programs, even if they think they are working well.

Retail giant Target, for example, had a mature threat hunting program, but the company decided it was time for a refresh to ensure the program was fit for purpose and still helping the business.

Evolution of a mature threat hunting program

Target’s threat hunting program had been in place for five years when it decided to do a “soup to nuts” reworking of the program, the company’s Principal Engineer of Cybersecurity David Bianco told attendees at the SANS Threat Hunting Summit in London last month.

“It was time to evolve that program into something more modern,” he said. “Not that there was anything wrong with it, but we had just had essentially the same program for several years and wanted to see if there were any updates that should be made.”

Bianco said that Target’s previous program was mature enough to be able to collect and use a high volume of quality security data from around the enterprise to create new analysis procedures. He labeled this a level 3 on Sqrrl’s Cyber Hunting Maturity Model. “That is a great place to be,” he said. “Most hunting teams would kill to be able to say on this model, ‘We are definitely a level 3’.”

The goal was to get Target to level 4 on that maturity scale and move beyond human-scale detection and drive more automation in its security processes. “Organizations at this level are able to not only successfully find those new incidents,” said Bianco, “but the key difference is they treat their threat hunting program more as a driver for the improvement of their automated detection.”

How to assess your threat hunting program

Before upgrading a threat hunting program, it’s important to assess what’s working and what isn’t. At the start of the refresh Bianco spoke to key stakeholders to get their perspectives of the program. He asked the people running the threat hunting program, the executives who were sponsoring it, as well as those receiving the briefings from the team what was and wasn’t working. He also sought to identify if there were any goals they wanted to accomplish that they previously hadn’t thought possible.

At the same time, Cat Self, lead information security analyst at Target, was brought in and wanted to gain insights into how the current program operated. She had similar roundtable discussions with the SOC analysts involved in the hunting to understand where they felt things were and weren’t working. She then joined the hunting operation to see the process firsthand.

“I sat in on a hunt so I could get a baseline of where we’re at and why we do what we do so that if we do change it, we know the cascading effect of those changes,” Self said. “There’s a lot of tribal knowledge inside of hunting that you don’t really see until you actually sit and participate in a hunt: Why that is our go-to tool? Why do we search environments using this method?”

From those discussions and observations, Target identified three areas to improve:

• Program focus: Realign the goals of the program with the company’s needs.
• Operational consistency: Ensure that the program runs as smoothly as possible on a recurring basis.
• Hunt topic strategy: Find more efficient ways to point the threat hunting program at specific problems that would benefit the company.

Step 1: Focus the goals

Target’s original threat hunting operation was set up to find security incidents that automated detection had missed, with a secondary goal of identifying gaps in visibility. Over time, that focus had shifted slightly and so needed realigning, says Bianco. “The program was run by our SOC folks, and although it [finding security incidents] was still a large component, their idea of it had morphed over time to be less about finding incidents and ensuring that we had proper visibility to using threat hunting as a mechanism for knowledge and skills transfer between the SOC analysts.”

During the refresh, the company wanted to move the program’s goals away from finding incidents or knowledge transfer. Although they were still important components, they would be by-products of the primary focus of enabling better automated detection.

“Our enterprise can’t rely on human-scale detection. We have to use humans to improve the automated scale detection,” says Bianco. “Our job is not just to find new security incidents. We will find those as a byproduct of doing our job. Our job is to produce prototypes, or proofs of concept, of new detection mechanisms or improvements in our existing technical detection mechanisms.”

Step 2: Ensure operational consistency 

Before the refresh Target had no dedicated, full-time threat hunters and relied on bringing in teams of SOC analysts who conducted week-long hunts on an eight-week rotation. While having a different set of analysts involved each time a new hunt started helped spread knowledge around the team, it often led to operational inefficiencies.

In addition to their full-time roles, the analysts working at Sqrrl’s level 3 and leading the hunt were expected on their rotation week to come in with a hunt topic and have all the necessary resources and information for the level 1 and 2 analysts ready ahead of time.

“What would happen is, sometimes the level 3 analyst didn’t have that prepared,” says Self. “Then we would be pulling the data on day one, formatting it day two, only really hunting on days three and four, and then presenting our findings on day five. We were asking too much our level 3s and of our rotational personnel to come in here and do all of this work.”

To remedy this, Target decided to create a dedicated threat hunting team, and the company now has three full-time threat hunters. The company also kept the idea of a rotating team of SOC analysts joining in on hunts to continue the knowledge transfer. The one-week cadence of hunts was also kept as the hunt team felt this would allow them to dive deep into individual topics and quickly pivot as required.  However, a two-week period was introduced between each week-long hunt to give the hunters more time to prepare, document findings, and follow up with other teams.

“We wanted to be able to, when we got in on day one, actually start hunting and hunt with reliability so our operations were a lot more predictable and have a reasonable expectation of how that flow would go,” said Self. “Hiring full-time threat hunters was a key element. That was really what allowed us to be able to execute every single thing we talked about: actually baking in your time in preparation, being able to create the full documentation of what happened and how to iterate on that next time, and giving our people time to actually prepare adequately for that next hunt.”

Step 3: Define the strategy

The third part of the program refresh was to change how Target picked topics to hunt on and add more strategy around prioritization. Previously the company’s L3 SOC analysts gathered in a room and floated ideas on a whiteboard. After discussions among the analysts, the chosen hunt topics were then scheduled on a calendar. While the analysts might be subject-matter experts, Self and Bianco say that the downside was these were often “a bunch of random ideas” and the company needed a better way to focus on what was important.

“We kept the idea that our subject-matter experts could contribute possible topics or hypotheses for us,” said Bianco. “We expanded that to explicitly solicit more ideas from more people in our extended security teams, but that idea basically still held.”

While they kept the collaborative nature of initial discussions, the company decided on new assessment criteria to prioritize which hunt topics posed the greatest risk to the business. The new assessment was based on three risk factors: prevalence of the proposed threat topic among Target’s most closely monitored threat actors, the prevalence of said threat across wider industries and infrastructures, and the business risk impact of said threat if it did hit the business. Each of those three factors was rated by the relevant team – threat intelligence, threat hunting and detection engineers respectively – and given a 0 to 5 scoring (0 being no risk, 5 being high risk).

Hunts are then grouped together in six- to eight-week sprints. Sometimes these sprints are themed – Target’s first hunt sprint was themed around host-based detection on Macs – and sometimes they are solely around priority scoring or special requests from the CISO or other executives if they required something specific be looked at. “That gives us a really good mix of potential long-term program improvement for cyber security,” says Bianco, “but also possibly things that are short term urgent for us to do hunting on.”

Self offers these key takeaways for companies looking to improve their current program:

• Hire full-time threat hunters.
• Give those hunters time to prepare before a hunt and document findings after a hunt.
• Ensure operational consistency and apply a proper strategy around what teams should be focusing on during hunts.