• United States



by Sean Martin

Drowning in security data? Here’s how to make threat intel work for you

Oct 19, 201512 mins
Big DataData and Information SecuritySecurity

drowning big data 2
Credit: Thinkstock

How does a company operationalize its risk and security programs? More specifically, with all of the talk about big data, how does a company operationalize its threat intelligence process?

Many companies think they know what the keys are to their kingdom and where the entry points are located. Unfortunately, they soon find out that the most serious breaches often take place somewhere else.

+ ALSO ON NETWORK WORLD: 5 tips for better enterprise security +

“Companies will watch their ATM activity and miss the subtle warning signs passing through their mainframe,” says Sharon Vardi, Chief Marketing Officer at Securonix. “Without knowing it, companies are leaving their crown jewels sitting exposed, prime for the taking.”

Knowing what to watch requires collecting data that can be analyzed, and allocating eyes to perform the analysis.

However, organizations won’t succeed if they don’t collect and analyze an ongoing full stream of data—success requires more than just a snapshot from a limited window of time. The data needs to be collected before, during and after malicious activity takes place.

“Companies also need to include data from inside and across the network, from each and every endpoint, and potentially even from external and public sources located outside of the network,” says Alan Hall, Director of Strategy & Product Marketing for the Advanced Threat Protection Group at Blue Coat Systems. “Otherwise, responses will be limited, at best.”

Effective incident response requires context

Being able to respond to an incident is where the rubber meets the road. This requires context—information beyond what is captured in raw form. Context can be used to identify an advanced or otherwise covert attack or compromise—plus give you the means to determine how best to react.

Travis Smith, Senior Security Research Engineer at Tripwire

“To properly manage security incidents, organizations not only need to collect data but also analyze the data in real time—and then store that data so it can be used later to correlate against new real-time data,” says Travis Smith, Senior Security Research Engineer at Tripwire. “The challenge is…storing data costs money—plus, the management and usage of the data can be a real problem as well.”

The reality is, security teams looking to analyze logs are at the mercy of the developers who decide what to log and from which systems. These details are often built into (or more accurately, excluded from) systems when they are developed.

Full packet capture uncovers the real meat

Even still, security logs are just the tip of the iceberg. The real meat is in full packet capture across the entire network. Getting past this log-only barrier and into network capture leaves companies with a load of security data—yet another challenge: “Security data isn’t big data,” says Smith. “It is morbidly-obese data.”

The normal best practice for data storage is 30 days of traffic, though some industry policies require more and some government regulations demand more. “It’s almost negligent if the security team is living purely in alert mode, unable to analyze context,” adds Hall.

Sometimes it’s more than just a matter of how much, it could also be an issue of how: customers seem to be struggling to get what they want from their security management programs. “Security teams either get no alerts or too few alerts…or they suffer from severe alert fatigue,” says John Humphreys, vice president of marketing at Proficio.

Other sources of data to consider

As Smith at Tripwire recommends, absolutely capture your log data but also look to move beyond logs and “organize some of your own internal network feeds. You should also tie sessions together to capture packet strings and ultimately perform a full packet capture.”

Vardi adds, “You should also consider external feeds of data that may not be traditionally categorized as security data.” This includes Facebook activity, job searches and other data sources available to you while your employees are operating from your corporate-owned and corporate-liable devices and networks.

“Open Source intelligence with company data is fair game in these circumstances,” adds Vardi. These data sources may not look like security data but can dramatically change the context of the security data and provide a new way for companies to look at their risk profile. Of course, to make threat intelligence useful, the feeds must be credible and based on reliable sources. This includes your own internal feeds. There are a large number of apps that generate a ton of seemingly-non-malicious internal traffic, most of which are designed to share data so the business teams can do their work. Still, the inclusion of these data feeds and the quality of this data can’t be overlooked.

These internal-only network communications are often dismissed or go undetected by only monitoring the intrusion and exfiltration-detection system logs. This is usually because the traffic moves horizontally within the network and never crosses the intrusion monitoring systems nor hits the path of the perimeter firewall.

“Intrusion and exfiltration only happen when the device traffic enters or leaves the corporate network,” says Carmine Clementelli, Sales and Marketing Manager at PFU Systems, a Fujitsu company. “Similarly the command-and-control communications happen off the network by using external, temporary websites. Most times, finding the problem at this layer means it’s too late.”

What context to look for?

When it comes to determining the context that will be used to seek out the threats an organization faces and the attacks they are experiencing, there are generally one of three options to consider:

  1. Let the systems define the context automatically and hope their vendor-defined configurations and rules “get it right.”
  2. Use your own learned context that you’ve garnered over time and hope that you know enough about your environment—or at least as much as the attackers know.
  3. Define the context in an on-the-fly, ad-hoc fashion; try to pull in the threat data and supporting intelligence to match; and then pray you can stay ahead of the game and not fall prey to alert fatigue.

Or you can take advantage of the security community and use cross-industry, cross-profile sets defined by others to select and then customize the context. “Security teams need to observe their IT life in reality by using other companies’ experiences,” says Humphreys. “This is a good way to understand the real context.”

When it comes to insiders stealing data and sending data to competitors, the context here lies in seeing your employees and contractors accessing data more frequently than is typical. You might also capture traffic that shows employees are sharing data outside the organization, such as via a personal email account or a removable USB drive.

An employee who recently had a poor review could be flagged as an even higher risk. If a third-party vendor makes multiple login attempts and attempts to access company systems not typically accessed, it could be a sign that the vendor is either acting maliciously or was hit by a phishing attack.

But it’s not just people and systems that provide context. “A document can be an entity as well,” says Vardi. “The behavior of a document is equally important to watch. Where does it live? Who accesses it? From which IP addresses is it accessed? Where does it travel to?”

Vardi adds, “Each of these—when viewed together along with other events and alerts—can bring additional context to otherwise undetected malicious activity. For example, if an employee, partner or customer usually logs in from a Windows PC using Firefox, and all of a sudden they download documents from a Mac using Safari, then that could be a sign of trouble brewing.”

ATM fraud is another real-world example that’s becoming a big deal these days. Picture a banking customer who has been with the bank for 20 years and interacts with the bank a certain way most of the time. You can look for anomalies in their activity: the amount of their withdrawals, the location of the withdrawals, and the card used. Perhaps even the number of times the card is used during the day at different locations.

You can use this same principle for monitoring access to business resources and other user and system activities on the network—not just ATMs and withdrawals.

Here are a few examples:

  • An endpoint allocated to a single user logs onto the network multiple times from the same location using multiple user-identities. If you see this, there’s a chance the system has been compromised.
  • Unencrypted North/South traffic correlated with internal East/West traffic—be on the lookout for network activity coming in and moving laterally. This connected flow could be the sign of an unauthorized user or device on the network.
  • Leverage behavioral-based detection techniques that look at outbound traffic and peer-to-peer traffic to see where the traffic is going and how frequently it travels that path. Focusing on the ingress shouldn’t be the only approach; you also need to assume the malware is already inside and monitor the egress.
  • Take advantage of command-and-control detection as well to identify existing attacks that are looking to exfiltrate data. Be aware that oftentimes the data exfiltration doesn’t come as a single download; it can happen as a series of small actions over a long period of time. What happens in the middle that represents the long period of activity are the lateral movements—based on behavior as opposed to just packet analysis. Consider the use of an IT/security approved website that’s been hijacked by the attacker as a storage service that won’t be detected by your reputation and filtering systems.
  • Go beyond top-level application monitoring to analyze application features that are being being used. Facebook as a whole may be deemed OK for some employees, but how and when are they using Facebook chat vs. Facebook video view vs. Facebook video upload? What and how much data is being transferred to/from each of these features?

How will you respond?

Context is not only necessary for detecting an attack, it is also paramount in identifying the source of the attack, blocking the spread of the attack, and fixing what’s been compromised due to the attack.

“With integrated detection investigation, analytics and forensics, you can see a zero-day alert on a network from four months ago,” says John Dasher, vice president of marketing at Niara. “You can then look at your logs, packet flow and threat feeds to associate a person with a given device. You can also see which users accessed certain systems, applications and documents to determine which resource—such as a PDF—was the cause of the harm.”

A sophisticated attack might look suspicious but may not generate an alert. But if there is an egress to a known bad IP address, you could see that the IP address that the PDF came from matches, and then take the appropriate action.

At the same time, it’s important to not get caught up in alert fatigue, where high volumes of alerts could lead to an investigation that may take days or weeks to close, forcing you to miss the real attack taking place somewhere else. You need to be able to link the activity to some context so you can act in the best way possible at the best time possible.

“The perfect storm isn’t always the case, and your network operations team doesn’t work 24/7,” Humphreys warns. “You should thus support the latest, most prominent firewalls and send a program command to block malicious traffic temporarily. You have to use the tools in context and in a smart, automated way.”

The value of operationalizing threat intelligence—in context

Many large companies position themselves as keepers of global intelligence because they have thousands of customers with tens of thousands of nodes, and they share the data with other companies. Ingesting and digesting this data and then relying on only signature-based and rules-based solutions means that constantly morphing malware can easily squeak by. “This isn’t operationalizing”, argues Clementelli.

As you plan to operationalize your threat intelligence program, keep in mind that the value of threat intel is only as good as the sources of the data and the programs the data feeds. A good analytics engine fed with bad data is not as good as a decent analytics engine with credible, relevant intelligence. Context has to be built around the other variables you can also see—security analysis requires more than pure security data.

When taking on this challenge, you will most likely need to identify and collaborate with a security expert trained in big data and security analytics. Similarly, be sure to identify solution providers and security vendors that can provide expertise in both internal and third-party vendor risk management as well as security incident response. It’s critical to thwart as many attacks as you possibly can up and down the supply chain, but when attacks succeed, it’s just as important to limit the damage and immediately return your network infrastructure to normal operations and to a fully secure state.

Sean Martin is a four-term CISSP and 25-year information technology and information security veteran. Write to him at