Bracing for the security data explosion

Organizations must prepare for collecting, processing, analyzing, and acting upon terabytes of security data.

data explosion / data streams / volume / velocity
SPainter VFX / Getty Images

CISOs should internalize this quote from the former Senator from Georgia, extrapolating its focus toward cybersecurity defense.  In other words, all decisions about cybersecurity strategies, program priorities, investments, etc. should be made based upon analysis of real-time and historical data.  What types of data?  EDR data, network meta data, cloud logs, identity data, threat intelligence, and so on. 

Some aspects of this data explosion are already happening.  ESG research indicates that:

  • 75% of enterprise organizations collect, process, and analyze more security data today than they did 2 years ago. Nearly one-third (32%) of organizations claim to collect, process, and analyze “substantially more” data than they did in 2018.
  • 52% of organizations retain security data online for longer periods of time than in the past while another 28% would like to retain security data online but can’t for cost or operational reasons.
  • To accommodate longer data retention periods, 83% of organizations use off-line or “cold” storage. This helps control infrastructure costs but makes retrospective investigations more cumbersome.

Growing security data analysis and operationalization requirements were already a priority at the beginning of 2020.  COVID-19 added to the urgency by introducing new data analytics use cases, traffic patterns, behavioral analytics needs, and blind spots. 

Once the summer ends, CISOs will start the planning process for 2021.  As they do, even smaller enterprises need to prepare for a quantum leap in security data collection, processing, and analysis requirements. 

Here are some thoughts around this transition:

  • CISOs should think about consolidated data management services — a repository for all security data regardless of its source, format, or type. While they are at it, they should caucus with CIOs to see if they can aggregate security and IT operational data into a common bucket.
  • In all industries and regardless of compliance requirements, the next iteration of security data collection, processing, and analysis will depend heavily on cloud-based resources. By 2022, most organizations will migrate all security data to the cloud or rely on hybrid architectures that are heavily weighted to cloud-based infrastructure.
  • Cloud resources will also be required for a wave of new security analytics at scale.
  • Security analytics and operations tools tend to focus on threat detection and response today. Look for a new round of innovation around big data analytics for cyber risk management — activities like attack surface management, third-party risk management, and vulnerability management that depend upon dynamic data collection and analysis.  Think in terms of a real-time CISO dashboard used for cyber-risk identification, prioritization, and mitigation.  Tools in this area come from companies like AttackIQ, Bugcrowd, CyCognito, Randori, etc.  FireEye certainly saw the intersection of security analytics/operations and cyber-risk management coming with its acquisition of Verodin. 
  • Security analytics require massive and unprecedented scale. We’ll see a steep increase in the use of MSSPs (AT&T, DeepWatch, Proficio, etc.) — even in the largest enterprises. Those that go it alone may need professional services help from ThetaPoint and others.
  • Security data pipeline expertise will be in high demand as organizations move to streaming data for real-time analytics. Since few security organizations employ data management engineers, professional and managed service providers will be called upon to bridge this gap.
  • We’ll see all types of security operations and analytics platform architectures (SOAPA) moving forward — marketplaces (a la CrowdStrike and PAN), partnerships (Google/Tanium, many Splunk partnerships, etc.), and tons of M&A activity.
  • As security data drifts to the cloud, cloud service providers (CSPs) like Amazon, Google, and Microsoft have a big home court advantage. This is one reason why all three have jumped into the security analytics and operations pool with Amazon Detective, Google Chronicle, and Microsoft Azure Sentinel.  To compete, other vendors (i.e., Devo, Exabeam, LogRhythm, Securonix, etc.) must out do native CSPs on ease-of-use, analytics, process automation, etc.
  • Related to my previous point, advanced analytics is a burgeoning battleground. This will bring data analytics specialists like Palantir, SAS, and others into the game.  It’s also why MicroFocus (ArcSight) acquired Interset and SumoLogic scooped up JASK.
  • Open source software like the ELK stack will play a role but most organizations won’t be able to program open source tools to keep up with the scale and dynamic nature of security analytics/operations needs. Commercial cloud-based solutions will own this market. 
  • I’m still unclear about the role of XDR, but it will remain a supporting technology initiative in the near future.
  • One interesting aspect of this trend is the abstraction and centralization of a security operations UI/UX. Some examples here are IBM’s Cloud Pak for Security and Splunk Mission Control. 
  • Finally, some people see these changes as a real threat to Splunk’s leadership position, but I don’t. Yes, Splunk will have to be agile to fend off new competitors and business models, but Splunk really gets this market and is investing and adjusting accordingly.

Lots of changes ahead.  Stay tuned, I’ll be following this progress carefully.

Copyright © 2020 IDG Communications, Inc.

7 hot cybersecurity trends (and 2 going cold)