Defining Big Data Security Analytics

At the end of 2012, ESG conducted a research project looking at big data security analytics from the demand-side. It turns out that market demand is already apparent — 44% of enterprise organizations consider their security analytics “big data” today, while another 44% believe that their security analytics requirements will be regarded as “big data” within the next two years. Okay, enterprise organizations need big data security analytics solutions today; but just what is a “big data security analytics” solution anyway? ESG just published a market landscape report to answer this very question by looking at the supply side to gauge existing solutions and future directions for big data security analytics. Defining big data security analytics has to start at a high level. Big data security analytics is simply a collection of security data sets so large and complex that it becomes difficult (or impossible) to process using on-hand database management tools or traditional security data processing applications. As the ESG demand-side data indicates, many enterprise organizations have already crossed this threshold as they collect but struggle to analyze multiple terabytes of security data. Big data security analytics solutions also distinguish themselves based upon three basic characteristics:• Scale. Big data security analytics solutions must have the ability to collect, process, and store terabytes to petabytes of data for an assortment of security analytics activities. • Analytical flexibility. Big data security analytics solutions must provide users with the ability to interact, query, and visualize this volume of data in an assortment of ways.• Performance. Big data security analytics must be built with an appropriate compute architecture to process data analytic algorithms and complex queries and then deliver results in an acceptable timeframe. In the early stages of this market, big data security analytics solutions are being developed and introduced along a continuum. There are two poles and thus two types of big data security analytics solutions that make up this scale:1. Real-time big data security analytics solutions2. Asymmetric big data security analytics solutionsReal-time big data security analytics solutions are actually an evolution of present day SIEM and log management solutions built for modern scale and performance requirements. These solutions are built around a distributed architecture; made up of appliances designed for local streaming processing and collective parallel processing. Real-time big data security analytics solutions tend to collect and analyze old standby data like logs, network flows, and IP packets across the enterprise with a view of the data from L2 through L7. Many of these solutions are based on some type of proprietary data repository as well. Examples of real-time big data security analytics solutions include Click Security, Lancope, and Solera Networks. Asymmetric big data security analytics is a relatively new category of solutions designed for the non-linear needs of security analysts who typically pivot from query to query as they investigate individual security events and/or anomalous behavior across systems, networks, user activity, etc. Asymmetric big data security analytics solutions can be built on proprietary data repositories, but it is likely that all products will support big data technologies like Cassandra, Hadoop, and NoSQL over time. Security analysts will feed these solutions with batch updates containing terabytes of structured and unstructured data in order to look at historical security trends over long periods of time. Asymmetric big data security solutions will be anchored by machine learning algorithms, cluster analysis, and advanced visualization. Early solutions in this area come from vendors like LexisNexis, PacketLoop, and RedLambda.Large enterprises need both real-time and asymmetric big data security analytics, although they may be able to cover some of their real-time needs with a few leading SIEM or log management solutions. A few vendors including IBM, LogRhythm, Narus, RSA, and Splunk offer capabilities in both areas, typically by integrating multiple products across a security data analytics architecture.Yes, the security analytics market is nothing new but new requirements for scale, performance, and analytics are making existing products obsolete. For this reason, I believe that the big data security analytics market will evolve and expand quickly in both real-time and asymmetric big data security analytics capabilities. So what’s the winning formula in this burgeoning market? Stay tuned to my blog as I’ll be discussing this soon.