• United States




Are you buried under your security data?

May 25, 20165 mins

Before you go looking for new sources of data, you'd better be able to use the data you already have.

In a good article this week, Ben Dickson at TechCrunch writes about the sharing of threat intelligence data between companies. I will confess upfront that I am skeptical about the whole idea of threat intelligence sharing. Companies may decline to share such information because they fear the loss of competitive advantage. Some who are otherwise agreeable will just decide they don’t have the time to be bothered.

As I see it, the big problem facing the security world today is not the lack of information, but rather the ability to process and act on it. Most of us find it a challenge to keep up with our email inboxes, let alone security logs and reports. If we cannot use our own data to advantage, why would we expect to be able to use one or more threat intelligence feeds?

This lack of ability to process our massive amount of security data probably contributes a great deal to the success of the pervasive security attacks many face. Hackers come to our networks to stay. They almost always leave telltale signs in some log or report, but unless you can centrally file, process and review your data, you will never know. Thus, many companies do not know they have been hacked until they find their data for sale on some dark website.

A good example of this is the growing ransomware epidemic. Although there is no reliable means of spotting ransomware infections before some damage is done, it is often possible to catch them and react before they impact a bunch of endpoints. Monitoring security data can be the key to containing this damage.

So, before we go looking for new sources of data, we’d better be able to use the data we already have. Sadly, this is hard. To fully review all logs and reports manually, a typical midsize company would need a bunch of employees staring at data for eight hours a day. Such positions are pretty hard to justify to company management. Further, the work is so boring that nobody would want the job anyway.

While it is not possible to eliminate all of the need for tedious data reviews, we can minimize this effort via the intelligent use of automation. The goal is to have some combination of software sifting through our massive amount of data, looking for things to bring to the attention of a human.

There has been much talk in the industry about using artificial intelligence technology to help with this task. I look forward to the availability of strong tools in this area. Right now, however, this field is in its infancy. We need to protect our systems and data in the here and now, so we must move forward with tools already available to accomplish this task.

Here are some thoughts on how to employ automation to dig through your massive pile of security data: 


The first step in automating your security data analysis is to get it all in one place. If you must look in 10 or 15 places for your data, analysis simply won’t happen. Fortunately, there are many good tools to address consolidation, including Graylog and Splunk, and Web-based services like Loggly

Decide what is important

Once your data is in one place, look at it carefully, and figure out what is important to you. I would suggest that well over 90% of log data consists of routine entries that are not very useful. You need to figure out how to spot the ones that are useful. 

Employ intelligent analysis 

All of the tools mentioned above include some form of data analysis automation. Armed with the knowledge of what is and is not important, set your system to only send you the important stuff. This will take some initial trial and effort, but will be well worth the time spent. 

When in doubt, review it 

It is not possible upfront to spot all of the records you need to know about. This sort of data, which occurs infrequently, may be critical to identifying an event. Make sure your system is set up to send you anything that it does not recognize. 

Keep a record 

When you find an anomaly in the records you review, add it to a log. This will ensure that a concern requiring further review does not fall through the cracks. In addition, a complete record of your prior investigations may help streamline future such investigations. 

Update your intelligence 

Many of the records not filtered by your logging and analysis product that show up in your inbox will ultimately not be of real interest. Make sure you update your automation to filter these records, so you don’t have to repeat the analysis the next time such records show up. 

For larger organizations that already have external threat intelligence feeds, consolidation packages such as LookingGlass’ ScoutPrime will consolidate threat intelligence details with selected log records, with some correlation options.

Bottom line: Your security data is critical, but it won’t accomplish anything if you cannot spot the significant records without looking at every one. Consolidating and using automation to review your security records will help turn your mountain of data into actionable intelligence.


Robert C. Covington, the "Go To Guy" for small and medium business security and compliance, is the founder and president of Mr. Covington has B.S. in Computer Science from the University of Miami, with over 30 years of experience in the technology sector, much of it at the senior management level. His functional experience includes major technology implementations, small and large-scale telecom implementation and support, and operations management, with emphasis on high-volume, mission critical environments. His expertise includes compliance, risk management, disaster recovery, information security and IT governance.

Mr. Covington began his Atlanta career with Digital Communications Associates (DCA), a large hardware/software manufacturer, in 1984. He worked at DCA for over 10 years, rising to the position of Director of MIS Operations. He managed the operation of a large 24x7 production data center, as well as the company’s product development data center and centralized test lab.

Mr. Covington also served as the Director of Information Technology for Innotrac, which was at the time one of the fastest growing companies in Atlanta, specializing in product fulfillment. Mr. Covington managed the IT function during a period when it grew from 5 employees to 55, and oversaw a complete replacement of the company’s systems, and the implementation of a world-class call center operation in less than 60 days.

Later, Mr. Covington was the Vice President of Information Systems for Teletrack, a national credit bureau, where he was responsible for information systems and operations, managing the replacement of the company’s complete software and database platform, and the addition of a redundant data center. Under Mr. Covington, the systems and related operations achieved SAS 70 Type II status, and received a high audit rating from the Federal Deposit Insurance Corporation and the Office of the Comptroller of the Currency.

Mr. Covington also served as Director of Information Technology at PowerPlan, a software company providing software for asset-intensive industries such as utilities and mining concerns, and integrating with ERP systems including SAP, Oracle Financials, and Lawson. During his tenure, he redesigned PowerPlan's IT infrastructure using a local/cloud hybrid model, implemented IT governance based on ITIT and COBIT, and managed the development of a new corporate headquarters.

Most recently, Mr. Covington, concerned about the growing risks facing small and medium business, and their lack of access to an experienced CIO, formed togoCIO, an organization focused on providing simple and affordable risk management and information security services.

Mr. Covington currently serves on the board of Act Together Ministries, a non-profit organization focused on helping disadvantaged children, and helping to strengthen families. He also leads technical ministries at ChristChurch Presbyterian. In his spare time, he enjoys hiking and biking.

The opinions expressed in this blog are those of Robert C. Covington and do not necessarily represent those of IDG Communications, Inc., its parent, subsidiary or affiliated companies.

More from this author