Review: Threat hunting turns the tables on attackers

Advanced Persistent Threats are able to slip past even the most cutting-edge security defenses thanks in large part to a diabolically clever strategy. The threat actors behind successful APTs research the employees, practices and defenses of the organizations they want to attack. They may try to breach the defenses hundreds or thousands of times, then learn from their mistakes, modify their behavior, and finally find a way to get in undetected.

Once a network is breached, most APTs go into a stealth mode. They move slowly, laterally compromising other systems and inching toward their goals. Post-mortems from successful attacks often show that the time an APT breached a system to the time it was detected could be anywhere from six months to a year or more. And, they are often only detected after making that final big move where there is a huge exfiltration of critical data.

But what if you could turn the tables on APTs? Instead of focusing on your perimeter defenses, what if you assumed that APTs were already hiding in your network and you launched software specifically designed to hunt down these active, but hidden threats before they can do real damage?

For this review, we tested threat hunting systems from Sqrrl, Endgame and Infocyte. Each program was tested in a large demo environment seeded with realistic APTs which had bypassed perimeter defenses and were hiding somewhere within the network of virtualized clients and servers. We also snuck active threats past perimeter defenses to see how these threat hunting programs detected, caught and killed the current breed of apex predators of the threat landscape.

We found that in order to deploy these products successfully, security professionals must change the way they normally think. These threat hunting tools are not the passive observers that we’re accustomed to, simply reacting to alerts triggered in the SIEM. Instead, these are aggressive hunters who prowl their own networks looking to prey on APTs and undetected malware.

Here are the individual reviews (also see screen shots of each product):

Sqrrl Data

The Threat Hunting Platform from Sqrrl Data was created by several ex-employees of the National Security Agency in 2012. Sqrrl integrates into any network and collects data from the SIEM as well as other sources, such as outside threat data feeds. It is normally installed as software but can be run in a virtualized or even a cloud environment.

Sqrrl does not install agents on endpoints, but can provide more information to hunters by incorporating data from existing endpoint protection programs.

The installation takes no more than a couple of hours for most deployments. Sqrrl offers one or two days of training as part of the installation process, though not a lot of time is needed due to the graphical and intuitive nature of the software. We became fairly proficient hunters and were able to track down leads and uncover hidden threats after only a few hours of instruction.

At the beginning of every day, security analysts are greeted with an overall control panel showing various indicators and suspicious behaviors along with their relative severity. Sqrrl needs about seven days examining user behavior before it can accurately predict the suspicious behavior component, and its machine learning ability makes it even more accurate over time.

It’s critical to note that the behaviors which bubble up to the Sqrrl dashboard are not ones that have triggered any type of SIEM alert. Anything over the threshold of potentially malicious behavior set by the network’s active security programs is handled by security personnel however they normally would do that. What is left are the odd little things that may, or may not, be an indicator of compromise which has slipped through the cracks.

Hunters can then use their expertise to investigate behaviors like beaconing, lateral movement, data staging, unusual usage patterns and exfiltration to create a hypothesis and potentially uncover a breach. It’s possible that hunters can also verify valid activities and clear them from further consideration.

+ ALSO: In depth: What does APT really mean? +

Since we knew that most APTs rely on privilege elevation as part of their pattern, we launched an investigation, or hunt, based on a single odd event captured by Sqrrl where an administrator logged into a system labeled C586. The strange thing was that the admin had never touched that system before, but since they logged on using valid credentials the first time they tried, no alerts were triggered. Sqrrl flagged the behavior, and thus we began our investigation.

The great thing about Sqrrl is that everything is displayed visually. We didn’t have to pore through the 85 pages of related log files, although they were available if we wanted, to find out what other systems had connected or were somehow involved with C586. We sent in a query from the drop-down menu and discovered a chain going back through four other systems with lateral movement ties to the one under investigation.

From there, we looked at beaconing behavior and discovered that the next system in the chain had beaconed out at some point over the previous month. Because beaconing behavior is one way that APTs reach back to their hosts, this was suspicious even though the IP was not one indicated as dangerous by threat intelligence feeds, and thus had not triggered any alarms.

Pushing our hunch, we searched for that IP address and were surprised to discover that two other systems in the same chain had also beaconed out to the same location. Now the picture was becoming more clear.

It also seemed like the fourth system in the chain, which had several denied access attempts recorded, was not actually part of the attack though it had connected with others that were. The failed access attempts were either the legitimate mistakes of a user forgetting their password, or perhaps deliberate camouflage from the attackers attempting to trigger an alarm to get security personnel looking in the wrong place.

Back to the three systems with beaconing behavior. We queried and found a rogue PHP process active on all three. Looking over time, it was clear that each system beaconed out and used that process only long enough to capture a new system before going dark. The attack chain finally stopped after it accessed C586, but didn’t install anything on it and did no beaconing from there.

A little while later, the administrator logged into C586 successfully even though they had never done so before. But using Sqrrl, we were able to discover why, and had a very good idea that those credentials were compromised, even though C586 was totally clean and triggered no alarms.

With a successful hunt completed, we could generate a report so that the network could be protected. The administrator’s compromised credentials could be rescinded as well as the login passwords for the compromised users. The IP of the beacon could be blocked and the PHP expunged so that the attackers will have wasted all that time only to be stopped short of their actual goal. And their tactics and techniques could be fed back into both Sqrrl and the network SIEM to catch them if they tried again using the same method.

The Sqrrl Threat Hunting Platform is a great tool to aid those hunting hidden threats inside their network. It works for users with any skill level, but more experienced analysts will be able to create better theories about attacks and thus likely have more successful hunts. Pricing for Sqrrl is based on the number of hunters who need to use the system and the amount of internal traffic data that needs to be analyzed. A system with a single hunter on a modest sized network would start at $25,000. Given that the average successful breach can cost half a million dollars or more in direct and indirect costs, sponsoring a hunter and equipping them with Sqrrl seems like a good preemptive investment.

Infocyte HUNT

Unlike the more traditional model of a lone hunter stalking their prey, Infocyte HUNT has added vast amounts of automation to the point where an entire network can be hunted in about a day. It’s more like hunting from a helicopter with a machine gun.

Founded by former Air Force officers in 2014, HUNT was designed to replace the sometimes months-long, labor intensive hunting process that some government agencies were using at the time. HUNT is completely centered on network endpoints and has no need for additional sensors. The main console, which is traditionally installed as a virtual machine, but is lightweight enough to exist on a laptop, sends out agents to all endpoints. However, the agents only exist for about 90 seconds on each endpoint and are dissolved afterwards. HUNT works natively with Linux and Windows endpoints plus most payment processing terminals. A Mac version is in the works.

Pushing out an agent takes up about 1 megabyte of network bandwidth while the return response is about 1.2 megs. The software defaults to sending out 60 at a time, and agents are smart enough to wait if the network is too busy, sending their report back when traffic clears. Using this method, HUNT is able to scan about 25,000 endpoints a day if the network is that large. Our test network had a modest 50 clients, so the total process took about a minute.

The main console controls the agent deployment and response process as well as the reporting dashboards, but heavy lifting is done in the Infocyte cloud. That includes hash and DNS lookups as well as comparing results with outside threat feeds and even sandboxing. Government agencies or companies that prefer to keep everything inside their networks can opt for a much larger on-premises configuration. In addition to the lookups, unknown executables can be submitted to Infocyte for analysis, where the staff maintains a threat lab to help identify zero-day type attacks. Human operators need to choose to submit those for analysis help, so again, data will only leave a network if it’s authorized to do so.

+ RELATED: Infocyte HUNT sets out to answer the question, “Have we been hacked?” +

To begin our investigation, we first had the console send out the dissolvable endpoints to our network. A report quickly came back because our test network was so small. From there, we could see that several endpoints could not be scanned. One of those had recently changed its login credentials. We could then log into it by hand and make sure it got the agent from the HUNT console. Another was disconnected from the network, so there was nothing we could do about that other than setting HUNT to catch it when it was back online. A couple of clients were VR machines that had been decommissioned but whose images remained in Active Directory. Those could be eliminated from future consideration.

The default scan looks at everything within the detection capabilities of HUNT including processes, modules, drivers, memory scanning, account information, network connections and hooks. Scans can also be tailored to specific items. If you are explicitly hunting malware disguised as a driver for example, you could just run that part of the scan. However, because the dissolvable agents are so quick, you don’t really save too much time paring them down, so the full scan is probably best most of the time.

With active endpoint scanning, HUNT could almost be deployed as a more traditional security tool, especially for organizations that have not invested heavily in endpoint protection. However, while HUNT can find traditional threats, its value as a threat hunting tool is that it is designed to catch advanced malware that would otherwise avoid detection.

As an example, in our testing we found an instance where Firefox.exe was listed as probably bad on one client machine. This was quite puzzling so we dove into that part of the report, which was easy to do using a good graphical interface. Drilling down to the first level, we found that everything with Firefox seemed fine. HUNT runs all endpoint programs through 21 anti-virus programs and provides a report back on their findings. In this case, all of them said that the file was fine, although HUNT was still not convinced. Drilling down further, the hash for the Firefox file was correct, so it was the actual Firefox program provided by the company.

We started to think that HUNT was providing us with a false positive, until we went a little deeper. It turns out that a module installed inside that version of Firefox turned out to be a bitcoin miner. HUNT not only caught this during the sandboxing process, but also allowed us to see every module that was part of the core program. That enabled us to identify a threat that would have escaped almost every other type of endpoint protection.

Another strength of HUNT is its ability to do true memory mapping, so malware that only exists within memory, even if it uses stealth technology or tries to stay under the radar, is quickly identified. We found a memory injection type of attack against Explorer on another machine. HUNT can take all types of memory code and convert it into executable files which can then have their characteristics checked in a safe environment. We have not previously seen this type of mapping process that allowed HUNT to drill so far down, and so accurately, into the system memory of connected endpoints.

Finally, HUNT is able to see if any type of malware or malicious process is using hooks to divert programs or users away from their intended destinations. On our test system, there was a program that was supposed to point to a specific place inside system memory, only a hook was being used to read from a different place each time the function call was made. That is a pretty subtle type of vulnerability that could be stealthily exploited by attackers without triggering too much attention. But HUNT found it and gave a detailed report about what was happening.

Once a scan is complete, a report with multiple levels can be generated. For the analysts, very detailed descriptions of all threats, where they reside and what they are attempting to accomplish is available. And for the C-suite, HUNT provides a really nice top-level overview of everything that is wrong or compromised within a network.

On the flip side, a HUNT report could also certify that a network is completely clean and uncompromised, something very few other programs are willing to do. A clean report shows everything that HUNT did and checked, and explains why it is so confident that no APT or other breaches exist. That should help executives sleep a little better at night.

Pricing for HUNT starts at $6,000 for 100 endpoint licenses with volume discounts available. Because of the way the scanning engine works with dissolvable agents, its scalability is practically unlimited. And with a constant rate of about 25,000 endpoints scanned per day, it’s easy to figure out how long a scan will take based on network size.

Infocyte’s HUNT would be a good program for organizations that are just starting to upgrade their cybersecurity defenses, particularly protecting endpoints. But it would also be a perfect check for organizations that have invested huge amounts of money in robust defenses. HUNT could check those cybersecurity programs and either point out any holes that still exist, or certify that those defenses are working perfectly.

Endgame

Founded in 2008, Endgame is one of the oldest companies in the threat hunting space. That makes sense given the maturity and feature set found in their signature product, which is also called Endgame. In fact, the company more clearly defines the role of a threat hunter in terms of someone who can prevent, detect and respond to threats, both known and unknown, and builds its tool around that philosophy.

The core Endgame console can be deployed as a virtual machine or placed on a physical system like an appliance. It can also exist in a cloud or hybrid environment. Once up and running, the program needs to deploy agents onto all the endpoints of the network that will be protected. The agents are powerful, able to work with Endgame to stop processes, delete files and restrict access to machines when needed. In a sense, the agents arm the hunters who will be prowling the network looking for threats. Not only can hunters find threats with Endgame, they can analyze and even destroy them.

+ RELATED: Accenture, Endgame team up to become the Van Helsing of cybersecurity +

Deploying the agents is a simple process using the included wizard. We first defined the IP range of our network and then Endgame was able to find every system and virtual machine in the test environment. Installation of the agents required the proper credentials, but was otherwise automatic. The agents work with any Windows system and any Linux desktop or server. There is no Mac version, though one is expected sometime next year.

Endgame does a lot to hide its presence in the network, ironically using a lot of the same stealth techniques as some APTs. There are no endgame exe files or directories on the system for example. The idea is not to alert a potential attacker that the protection exists should they get a peek at what is resident on the endpoint they are attempting to compromise. There is also a tripwire and tampering function on each agent that automatically alerts the main console should anyone try to manipulate or disable the agent.

One interesting aspect of Endgame is that in addition to providing the tools for threat hunters, it also acts as a more traditional endpoint protection program that fills the same role as antivirus, stopping all the low-level threats automatically so that they never bother analysts.

In our testing, Endgame worked in conjunction with antivirus programs already installed on the endpoints, though they do sometimes compete to be the first to stop a threat. If the antivirus intercepted the threat, it never got to Endgame. Likewise, if Endgame grabbed it first, the antivirus never triggered.

None of that really matters in terms of hunting because the caught threat never enters the realm of the hunter. But it does show that Endgame can either fill the role as the primary antivirus protection for endpoints or work alongside whatever program is already being used as an extra layer of security.

The threat hunting main interface provides a clean list of all endpoints being protected within a network, their IP addresses, what OS is running, how long the asset has been active, plus any alerts the machines are generating. Like with most threat hunting tools, the alerts within the Endgame console are ones that have bypassed traditional protections and are likely unknown to the SIEM. Endgame simply takes those odd little events and compiles them into an alert type format that is well known to most analysts.

Hunters can either move through the collected alerts or start their own investigation in the Hunt Selection menu. Using that wizard, threat scans can be triggered based on a variety of factors like loaded drivers, processes, removable media, system configurations, user behavior or other factors, or a full scan can be triggered to look for everything.

Drilling down into an alert with Endgame is a surprisingly easy process that walks hunters through the process and provides helpful advice on what to look for, and what actions to take. For example, when we triggered a Persistence type hunt looking for APT indicators, we found a system that might have been COM hijacked. In addition to all the file and path information, Endgame also explained what type of potential attack we were examining, saying in part, “In a Component Object Model hijack, the adversary writes a current user COM entry in the registry corresponding to a legitimate entry in the local machine hive. Because user objects override local machine objects, the operating system loads the malicious code when the component is executed instead of the legitimate code.”

It also offered advice on what we should be specifically looking for in our hunt and what responses we should take, directing us to that part of the program. Having that level of support was like having a trusted guide along on our hunt, reading the land and directing us to places where our efforts might find more success.

It even went so far as to identify the graph we were looking at, and suggest nine other interesting analytical views that it could generate to help get a better picture of the potential threat. In this way, Endgame would work extremely well as an advanced tool for an entry-level analyst, sort of training them along the way, and for a tier-three professional, who could simply choose to ignore the advice if they wanted.

We confirmed the fact that a COM hijack was in play and that it had not been detected by other network protection. At that point, we could generate the SHA 256, MD5 and SHA1 for the process in question so that it could be searched for on other endpoints. We also had the ability to get the file itself for further analysis or sandboxing and to kill the process on the endpoint in question. Being able to kill the process and delete the file is an incredible tool that would allow hunters to not only stalk their prey with Endgame, but also to take a shot at fixing the problem right there on the spot.

To fully test Endgame’s capabilities, the same type of attack that was used in the infamous Carbanak breaches was deployed against the test network. That attack got through the defenses of many major banks and was responsible for the theft of up to a billion dollars. Delivered as a spear phishing attack, it was unique in that the attack code was tailored for each bank it hit, so no signature-based protection worked. It was also very advanced because it injected malicious code into a real file, which then jumped into system memory, the only place it existed. It did not even change the MD5 hash of the compromised file, so bypassed almost all protections at the time.

In our test network, it also got around the perimeter defenses. However, it was detected as an anomaly by Endgame. In our hunt, we could fully scan and evaluate the memory of the compromised machine. Endgame identified the originally infected file, a common print server exe used in Windows systems, and the associated modified memory header, complete with the location of every changed memory address. The total size of the malicious code was only 36.9 kbytes, and only existed in memory, yet could have been responsible for the theft of a billion dollars. We found it on our hunt using the Endgame tools, and could have stopped it right then and there.

In terms of pricing, Endgame starts at $225,000 for 5,000 endpoints for an annual subscription and premium support. There are also multiple configurations based on subscription models and the number of endpoints to be protected.

Endgame would be a perfect solution for an organization that was moderately mature in terms of cybersecurity protection, but didn’t have a threat hunting capability. The program does a great job of helping tier one and two analysts become effective hunters alongside their more experienced coworkers, a huge benefit in this era of IT expert shortages.

It’s also the most complete hunting package that we looked at, offering endpoint protection as a standalone service or in conjunction with antivirus as a valuable extra. And it actually loads hunters up with the weapons needed to not only detect threats that slip past traditional defenses, but also to kill them – the mark of a true hunter.

Breeden is an award-winning reviewer and public speaker with over 20 years of experience. He is currently the CEO of the Tech Writers Bureau, a group of influential journalists and writers who work in government and other circles. He can be reached at jbreeden@techwritersbureau.com.