Log management is leading use case for Big Data

Companies that have deployed Big Data solutions are most likely to be using them for log management, according to a study released last week by the SANS Institute. This security use case was followed closely by data archiving, operational data storage, advanced analytics, data discovery, and search.

According to the survey of more than 200 professionals in IT, security and compliance, more than half of the 55 percent of organizations that have deployed Big Data projects use the technology for log management. In addition, of those who plan to deploy Big Data in the next two years, 58 percent said that log management is a priority.

Log data, intrusion alerts, and other types of security-related information is a perfect fit for Big Data systems, said Sam Heywood, director of the Cloudera Security Center of Excellence at Cloudera, which sponsored the report.

The volumes are large, the information comes in a variety of data types, and it’s coming in at a high velocity.

“We’re talking to organizations processing billions of security events a day,” he said. “And you’re pulling in data from multiple endpoints on your sensor grid, all the log files are going to look different, and any one system will change the data it’s sending back as firmware updates happen.”

Traditional approaches to collecting this information mean that analysts have to spend days, or even weeks, compiling data before they can begin to analyze a security incident.

Big Data not only offers a cost-effective, scalable platform to collect this data, but also provides analytics tools to look for long-term and subtle patterns that might be undetected by traditional rules-based and signature-based approaches.

“We knew the use of Big Data in cybersecurity was big,” said Heywood. “But we didn’t know it was this big.”

The survey also asked about the types of data that were being stored in the Big Data systems, and much of it turned out to be sensitive data of one form or another.

Of the companies that had deployed Big Data technology, 73 percent used it to store personally identifiable information, 64 percent for employee records, 59 percent for intellectual property, 53 percent for payment card information, and 40 percent for national security intelligence data.

The study shows the level of trust that companies are starting to put in their Big Data platforms, said Heywood.

[ ALSO: The 5 worst Big Data privacy risks (and how to guard against them) ]

“Big Data has arrived,” he said. “People are using it for mission-critical applications.”

The next steps that companies are taking involve increasing security. For example, fewer than 25 percent of companies currently use tokenization or encryption — but between 25 and 31 percent of companies plan to roll out some form of tokenization or encryption in the next 12 months.

When it comes to access control, about 45 percent of companies with Big Data projects have role-based access and around 16 percent plan to add it in the next 12 months.

“Security is front-of-mind for a lot of organizations,” said Heywood. “They want to make sure they’re operating their Big Data systems in a secure and compliant fashion.”