Is Signature- and Rule-Based Intrusion Detection Sufficient?

Signatures and rules are the bulwark of traditional intrusion detection systems (IDS), however they are also a significant source of frustration. Most IDS systems are known for being noisy, generating too many false positives and for being expensive to buy and manage – yet they remain a critical part of most organizations security stacks. IDS products require significant investments in hardware, configuration and tuning, both at initial setup and for ongoing maintenance. Threats are increasing, myopic point products are multiplying and traffic continues to increase exponentially, (22 percent CAGR in annual IP traffic, which will grow from 1.1ZB in 2016 to 2.3ZB in 2020), which means traditional intrusion detection approaches will simply generate more alarms. Short-staffed security teams are already so overwhelmed with excessive alerting that they don’t know where to start. What they need is an obvious way to prioritize investigations so that their efforts will be more beneficial to their organizations.The SANS Institute estimates that automated tuning could save $39,720 annually for a 7,500 person enterprise with 10,000 nodes. Next generation intrusion detection systems must provide automated tuning, event correlation and easy deployment models, to help to soothe this sources of frustration and reduce the costs associated with IDS deployments.

While automated tuning is one important improvement, it still doesn’t address inherent problems with rules and signatures: they can’t detect unknown attacks and are only given one chance, at line speed, to identify if packets are good or bad. In many multi-stage advanced attacks the attackers don’t always use immediately identifiable methods to compromise perimeter defenses, and often easily bypass the IDS to gain access to the network. For example, an attacker may use social engineering to discover personal emails of employees, and use those in a phishing attack which results in some of those employees’ personally-owned devices being infected. When those employees use the compromised devices within the corporate network, the infection spreads and eventually results in a breach of sensitive corporate information.

Sometimes the steps of a multi-stage attack can be identified given enough time and resources to conduct an exhaustive investigation, but typically these investigations lead to incomplete conclusions due to a lack of all the necessary data. Organizations want intrusions detected as quickly as possible, so time is a luxury not afforded to the security team. Traditional IDS captures only the traffic that matches a signature or rule, which is nowhere near enough to create the complete forensic picture of exactly what happened.

I’m in no way discounting the importance of signatures and rules. They are and should continue to be an important part of any comprehensive security strategy. However a new approach is needed that detects both known and unknown attacks with equal efficacy, in a way that doesn’t increase the noise and generate more false positives.

Data science/machine learning is the new approach du jour. However unlike other fads, it will have staying power only if done correctly. Providing machine learning models with enough training data to ensure effective detection (i.e., low false positives) is a known challenge. Fortunately, we are seeing more big data solutions coming on to the market with a large enough dataset to address this issue. However, much behind the scenes manipulations are required to make this network traffic useful, such as skillfully extracting the appropriate features so that the models aren’t viewed as another source of spurious alarms.

The concept of machine learning is not new but one of the reasons it hasn’t been more broadly adopted is because it requires intense processing power and access to data for training the various models for accuracy. I’m not talking about running the data through a single model. It must go through multiple. This multifaceted analysis is needed to detect suspicious actions hidden in network traffic as attackers get very creative as they try to cover their tracks. Then any relationships between seemingly unrelated actions must also be deduced so that when an analyst is alerted to a security event, he can immediately see the entire chain of suspicious actions that contributed to that event and understand why it was generated. And all of this must happen in the now. Just because it’s an unknown attack hasn’t changed what is a constant desire at all organizations: real-time detection.

This brings me to one more last requirement for the success of any new approach to intrusion detection: time. By that I mean the ability to go back in time as far as needed – a week, a month, a year or more – to reconstruct the network and reanalyze it for signs of attacks for which there were no previous indicators. Learning about a threat that has slipped past your defenses allows you to comply with any breach notification laws and put measures in place to prevent it from happening again. Given that multi-stage attacks unfold slowly, bringing time into the equation enables you to find and halt an attack earlier in the Cyber Kill Chain, have access to all of the forensic data and therefore understand the impact and limit the damage to your organization.

Real-time signature- and rule-based detections alone are no longer sufficient for intrusion detection. A multi-pronged approach that marries signatures, threat intelligence, machine learning and anomaly detection, coupled with automated retrospection and correlation, is what is required to produce the highly reliable detection that security analysts need. Correctly doing so is a non-trivial hurdle, one that most solution providers cannot overcome. Those that do are providing their organizations with a key element that is often missed when building a comprehensive security strategy.