It’s time to think harder about security data management

Organizations are running into scalability problems as they collect, process, and analyze more security data. It’s time for enterprise security data management.

security data management
Getty Images

According to ESG research, 28 percent of enterprise organizations collect, process, and analyze substantially more security data then they did two years ago. While another 49 percent collect, process, and analyze somewhat more data than they did in the past. (Note: I am an ESG employee.) 

What’s happening here? Well, first of all, organizations are collecting more data from traditional sources — system logs, vulnerability scans, network flows, etc. They are also grabbing security data from supplementary security sources, such as EDR tools, behavioral analytics systems, threat intelligence feeds, etc. Oh, and over the past few years, enterprises started gathering data from IoT devices, public cloud services, SaaS, etc. It all adds up to a growing pile of terabytes of security data.

In theory, all this data acquisition is a good thing. Armed with gobs of security telemetry, security analysts can assess events, prioritize actions, and make decisions based upon real-world activities.

Unfortunately, many CISOs I speak with are running face first into a security data scalability wall. Collecting, processing, and storing terabytes of data costs a lot of money, while managing growing volumes of data isn’t easy. Security professionals know the answers to their questions are out there, but they lack the skills, processes, and infrastructure to find them.

In my humble opinion, too many organizations made the tactical decision that more security data was a good thing and subsequently piled more and more data on the SOC. OK, but now security operations teams are buried in this very data. Ironically, this results in longer threat detection and decision cycles.

How to handle the growing security data 

What can be done here? Leading organizations are addressing growing security data volumes by doing the following:

  • Building a security operations and analytics platform architecture (SOAPA). The foundation of SOAPA is a distributed data management layer, which is a common security data repository built for massive scale that provides data access to security analytics tools. Distributed data management is well understood by technology vendors such as Oracle and SAP, but it is only starting to appear in cybersecurity. This is one reason why enterprise data analytics pros like the SAS Institute have been pushing their way into cybersecurity.
  • Aligning data sources with processes. Rather than trying to plough through piles of data, many organizations are letting security operations processes guide them to what’s most important. This is where security operations automation and orchestration tools play a starring role, as they can be used to align workflows with data curation, contextualization, and distribution.
  • Moving security data to the cloud. Yeah, I know security data is sensitive, causing many security pros to eschew cloud-based solutions. Well, everything in life is a tradeoff, so it may be worthwhile to consider moving security data to the cloud in lieu of the capital and operating cost and complexity of keeping in onsite. SumoLogic has long offered a SIEM in the cloud solution, while IBM and Splunk can be deployed in the cloud now, as well. Meanwhile, CrowdStrike, FireEye, Kenna Security, and Palo Alto Networks all utilize the cloud as part of their security operations offerings.
  • Letting machines do the work for them. Let’s face it, security data volumes have grown well beyond human ability to consume them. This is one reason why artificial intelligence, cognitive computing, and machine learning are an inextricable part of security operation’s future. 

I agree that cybersecurity should be based upon data-driven decisions, but few organizations have the resources or chops of Facebook, Goldman Sachs, or the NSA to cope with a growing security data tsunami. Collecting more data was the easy part; now we must become much smarter about how we manage all this security data more effectively. 

SUBSCRIBE! Get the best of CSO delivered to your email inbox.