Real-Time Big Data Security Analytics for Incident Detection

I’ve spent the last year or so doing research on the burgeoning field of big data security analytics. Based upon the time I’ve spent on this topic, I’m convinced that CISOs are looking for immediate help with incident detection, so they will likely focus on real-time big data analytics investments in 2014.

What do I mean by real-time big data security analytics? Think stream processing of data packets, network flows, and Meta data looking for anomalous/suspicious network activities that provide strong indication of a security incident in progress. A multitude of vendors, including ISC8, 21CT, Click Security, Hexis Cyber Solutions, IBM, Lancope, Leidos, LogRhythm, Netskope, RSA Security, and Solera Networks (and others), play in this space.

OK, real-time big data analytics is focused on incident detection, but what problems are CISOs really looking to solve in this area? Some recent ESG research may provide a few answers. ESG asked 257 security professionals working at enterprise organizations (i.e. more than 1,000 employees) to rank their security team’s incident detection challenges. These responses shed light on requirements for real-time big data security analytics:

• 39% of organizations say they are challenged by, “a lack of adequate staffing in security operations/incident response teams.” Real-time big data security analytics tools must be able to make existing staff more efficient and productive to overcome this limitation.
• 35% of organizations say they are challenged by, “too many false positive alerts.” Real-time big data security analytics tools must use streaming processing, advanced intelligence, algorithms, and visual analytics to filter out the noise and pinpoint problems with strong accuracy. 
• 29% of organizations say they are challenged because “incident detection involves too many manual processes.” This one is tough because security analysts pride themselves on their ability to spot an anomaly and pivot from data point to data point to find real problems. While this behavior is certainly admirable, it just doesn’t scale. Real-time big data security analytics tools should support analyst’s standard operating procedure but also help them automate investigations and forensic collection. This could be the catalyst to finally change security analysis from art to science.
• 29% of organizations say they are challenged because “incident detection depends upon too many independent tools that aren’t integrated together.” So real-time big data security analytics must supersede this army of point tools (with advanced functionality and integration). To paraphrase Harry Truman, the (incident detection) buck stops here.

Clearly, real-time big data security analytics vendors must be able to address these existing challenges. Those best able to do this will likely have a very Happy New Year.

On the flip side, CISOs should use this data as a requirements list for product evaluations and proof-of-concept trials. In other words, judge real-time big data security analytics tools on their incident detection acceleration AND their ability to improve staff efficiency, reduce false positives, automate manual processes, and supersede point tools. Smart CISOs will define metrics to gauge improvement in each of these areas over the short- and long-term.