Big data security analytics mantra: Collect and analyze everything

In a recent research survey, ESG asked security professionals to identify the most important type of data for use in malware detection and analysis (note: I am an employee of ESG). The responses were as follows:

• 42% of security professionals said, “Firewall logs”
• 28% of security professionals said, “IDS/IPS alerts”
• 27% of security professionals said. “PC/laptop forensic data”
• 23% of security professionals said, “IP packet capture”
• 22% of security professionals said, “Server logs”

I understand this hierarchy from a historical perspective, but I contend that this list is no longer appropriate for several reasons. First of all, it is skewed toward the network perimeter which no longer makes sense in a mobile device/mobile user world. Second, it appears rooted in SIEM technology which was OK a few years ago, but we no longer want security technologies mandating what types of data we can and cannot collect and analyze.

Finally, this list has “old school” written all over it. We used to be limited by analytics platforms and the cost of storage, but this is no longer the case. Big data, cheap storage, and cloud-based storage services have altered the rules of the games from an analytics and economics perspective. The new mantra for security analytics should be, “collect and analyze everything.”

What makes up “everything?” Meta data, security intelligence, identity information, transactions, emails, physical security systems – everything!

Now, I know what you are thinking:

1. I don’t have the right tools to analyze “everything.” You are probably right, but this situation is changing rapidly. Network forensic tools from Blue Coat (Solera Networks), Click Security and LogRythm can perform stream processing on network packets. Big data security analytics platforms from IBM, Leidos, Narus, RSA Security, and Splunk are designed to capture and analyze structured and unstructured data. Heck, there are even managed services from Arbor Networks and Dell if you don’t want to get your hands dirty.
2. I don’t have the skills to analyze “everything.” Very good point, and things aren’t likely to improve – there’s a global cybersecurity skills shortage and more data to analyze each day. Security analytics vendors need to do a better job here in terms of algorithms, automation, dashboards, machine learning, and threat intelligence integration. While I expect a lot of innovation in this area, CISOs should take a prudent approach here. For example, Splunk customers talk about collecting the data, learning the relationships between events, and then contextualizing specific data views by creating numerous dashboards. Makes sense to me.
3. I can’t afford yottabytes of storage for all of this data. With the exception of the NSA and its Bluffdale Utah data center, few organizations do. To be clear, big data security analytics doesn’t demand retention of data, but it does demand scanning the data in search of suspicious/anomalous behavior. In many cases, CISOs only retain the Meta data, a fraction of the whole enchilada.

While it may seem like hype to our cynical cybersecurity community, big data is radically changing the way we look at the world we live in. For example, we no longer have to rely on data sampling and historical analysis, we can now collect and analyze volumes of data in real time. The sooner we incorporate this new reality into our cybersecurity strategies, the better.