Are you buried under your security data?

In a good article this week, Ben Dickson at TechCrunch writes about the sharing of threat intelligence data between companies. I will confess upfront that I am skeptical about the whole idea of threat intelligence sharing. Companies may decline to share such information because they fear the loss of competitive advantage. Some who are otherwise agreeable will just decide they don’t have the time to be bothered.

As I see it, the big problem facing the security world today is not the lack of information, but rather the ability to process and act on it. Most of us find it a challenge to keep up with our email inboxes, let alone security logs and reports. If we cannot use our own data to advantage, why would we expect to be able to use one or more threat intelligence feeds?

This lack of ability to process our massive amount of security data probably contributes a great deal to the success of the pervasive security attacks many face. Hackers come to our networks to stay. They almost always leave telltale signs in some log or report, but unless you can centrally file, process and review your data, you will never know. Thus, many companies do not know they have been hacked until they find their data for sale on some dark website.

A good example of this is the growing ransomware epidemic. Although there is no reliable means of spotting ransomware infections before some damage is done, it is often possible to catch them and react before they impact a bunch of endpoints. Monitoring security data can be the key to containing this damage.

So, before we go looking for new sources of data, we’d better be able to use the data we already have. Sadly, this is hard. To fully review all logs and reports manually, a typical midsize company would need a bunch of employees staring at data for eight hours a day. Such positions are pretty hard to justify to company management. Further, the work is so boring that nobody would want the job anyway.

While it is not possible to eliminate all of the need for tedious data reviews, we can minimize this effort via the intelligent use of automation. The goal is to have some combination of software sifting through our massive amount of data, looking for things to bring to the attention of a human.

There has been much talk in the industry about using artificial intelligence technology to help with this task. I look forward to the availability of strong tools in this area. Right now, however, this field is in its infancy. We need to protect our systems and data in the here and now, so we must move forward with tools already available to accomplish this task.

Here are some thoughts on how to employ automation to dig through your massive pile of security data: 

Consolidate

The first step in automating your security data analysis is to get it all in one place. If you must look in 10 or 15 places for your data, analysis simply won’t happen. Fortunately, there are many good tools to address consolidation, including Graylog and Splunk, and Web-based services like Loggly. 

Decide what is important

Once your data is in one place, look at it carefully, and figure out what is important to you. I would suggest that well over 90% of log data consists of routine entries that are not very useful. You need to figure out how to spot the ones that are useful. 

Employ intelligent analysis 

All of the tools mentioned above include some form of data analysis automation. Armed with the knowledge of what is and is not important, set your system to only send you the important stuff. This will take some initial trial and effort, but will be well worth the time spent. 

When in doubt, review it 

It is not possible upfront to spot all of the records you need to know about. This sort of data, which occurs infrequently, may be critical to identifying an event. Make sure your system is set up to send you anything that it does not recognize. 

Keep a record 

When you find an anomaly in the records you review, add it to a log. This will ensure that a concern requiring further review does not fall through the cracks. In addition, a complete record of your prior investigations may help streamline future such investigations. 

Update your intelligence 

Many of the records not filtered by your logging and analysis product that show up in your inbox will ultimately not be of real interest. Make sure you update your automation to filter these records, so you don’t have to repeat the analysis the next time such records show up. 

For larger organizations that already have external threat intelligence feeds, consolidation packages such as LookingGlass’ ScoutPrime will consolidate threat intelligence details with selected log records, with some correlation options.

Bottom line: Your security data is critical, but it won’t accomplish anything if you cannot spot the significant records without looking at every one. Consolidating and using automation to review your security records will help turn your mountain of data into actionable intelligence.