Security Analytics Is In Fact A Big Data Problem

At the 2012 RSA Security Conference, ESG participated on a panel of experts to discuss whether security analytics would soon be considered “big data.” ESG and the other panelists agreed that this intersection was coming, we just weren’t sure of the timing.Were we right? ESG recently completed research project to find out. As part of this effort, ESG surveyed 257 security professionals working at enterprise organizations (i.e. more than 1,000 employees) were provided with the following definition of “big data:”ESG considers data to be big once the volume exceeds the capability and boundaries of traditional IT infrastructure. Difficulties include capture, storage, search, sharing, analysis, and visualization. “Big data” requires engineers to rethink and possibly redesign architectures such that they can support business requirements as data volumes grow. When applied to analytics, big data can also be characterized by the speed with which organizations require data processing, data integration, and data analytics tasks be completed in order to spot business trends, prevent diseases, combat crime, etc. Once presented with this description, security professionals were asked this simple question: Based upon this definition, do you believe that security data collection and analysis would be considered “big data” at your organization? The result? The RSA panel was right about the trend but overly conservative about the timing. Forty-four percent of enterprise security professionals believe that security data collection and analysis would be considered “big data” at their organizations today while another 44% believe that security data collection and analysis will become “big data” at their organizations within the next 24 months. To be clear, this does not mean that CISOs are actively hiring data scientists, implementing Hadoop, and sending CISSPs out for training on Cassandra, Hive, MapReduce, or Pig. It does indicate however that they are collecting massive amounts of data and existing security analytics tools can no longer keep up. As a result, IT risk continues to increase – a very scary scenario.CISOs may not have the time to piece together big data security analytics solutions but security vendors should be all over this! Some already are: IBM, SAIC, and Splunk for example. Others will follow suit soon. Security analytics is getting harder and harder so we need new tools, services, and expertise. As the ESG data indicates, we no longer have to postulate that this will happen in the future – it’s happening now. For users, this means massive near-term changes to their security infrastructure, investment, and organizations. For vendors it means new lucrative market opportunities beginning immediately. Finally, those rare security professionals proficient in mathematics and statistics will be sitting in the cat-bird seat for rapid career advancement.