Some Big Data security points to chew on

In preparation for the Big Data security panel I'm moderating at RSA, I asked participants for some early feedback. What I got back is worth sharing now.

What follows are two pro-con lists I'm incorporating into the PowerPoint slides to help drive the discussion. This is everyone's chance to chime in. I'll take the feedback I get and use it in the panel discussion. Don't be shy.

First, some pros and cons sent to me by Andrew Jaquith, CTO of Perimeter E-Security:

Advantages of using Big Data for security

1. Natural affinity for huge quantities of "machined" security data (e.g., we filter through 450m events per day)

2. Low barrier to experimentation

3. Works well for data that must be post-processed or refined (map/reduce)

4. Encourages exploration

5. Well suited for MSSPs and other companies who handle large amounts of customer security information


1. Analysis tools still very immature

2. Impedence mismatch between key/value and relational storage

3. High-skill analysts are hard to find

4. Need to know the questions to ask before you build out

5. Yet another analysis paradigm

And now this list from some friends of mine at Sourcefire (I ran it in another post two days ago, but I thought it would be useful to present next to Andrew's list for easy comparison):


•Security is often about detecting anomalies, and to do so, you need to have a full spectrum view that you typically can only get if you have enough data to know what constitutes “normal” versus “abnormal”.

•The goal with many information security solutions is to translate “back office intelligence” into “customer facing protection”. In recent years, the amount of back-office intelligence security firms are dealing with has grown tremendously (e.g., growth of malware samples, large volumes of sensor data, etc.). Big data techniques lend themselves nicely to this domain.

•To make the most accurate (security) decisions, we need to take advantage of all the intelligence available to us – from sensors, logs, user activity, etc. Big data techniques can be used to extract the most value from this wealth of information.

•Big data techniques are also useful in doing more broad visualization of security-related metrics. Having such a big picture understanding can help identify root causes to problems. In contrast, many “traditional” approaches only address symptoms rather than causes.

•Big data techniques can lead to entirely new sets of security capabilities. For example, in Sourcefire’s case, retrospective threat detection fundamentally leverages big data techniques. We are likely just scratching the surface here, and there are a wealth of new opportunities waiting to be uncovered.


•While there has been a rapid proliferation of “big data” technologies out there, not all of them are well baked enough to be used in production environments.

•Security decision-making needs to be rapid, and that does not always align with the batch-oriented processing of large data sets.

•There are no one-size fits all big data technologies. You have to understand both the problem you are trying to solve and the technology you are thinking of leveraging to solve it. If you aren’t sufficiently familiar with one or the other, there is a good chance your approach will ultimately prove fruitless.

•When you have a powerful hammer, everything starts to look like a nail. Big data techniques are powerful, but not every security-related problem requires them, nor can they magically solve every problem that comes up. Instead, it’s important to apply domain expertise and common sense first.

•Before focusing on “big data”, focus on “good data”. Many people try to apply sophisticated data mining techniques, but on data that might be dubious or otherwise poorly collected. For all their merits, data mining techniques are very much “garbage-in-garbage-out.”

The panel thus far includes Adam O'Donnell from Sourcefire, Andy Jaquith from Perimeter Security, Rich Mogull from Securosis and John Adams from Twitter. We're scheduled for Feb. 28 at 3:50 p.m San Francisco time.

--Bill Brenner

one-stop view of latest business threats. We created it for you! Bookmark it! Use it!

CSO's Daily Dashboard gives you a

Copyright © 2011 IDG Communications, Inc.

Subscribe today! Get the best in cybersecurity, delivered to your inbox.