• United States




Forewarned is forearmed: Using log management to prevent data breaches

Nov 11, 20155 mins
Data and Information SecuritySecurity

The importance of good log management to preventing data breaches

error logs monitor
Credit: Thinkstock

Readers of my blog will certainly be aware of the importance I place on the collection and handling of system logs. These logs contain critical data related to what is happening to your systems and networks that is not readily obvious, not the least of which are indicators that your network is being probed by potential hackers. 

Proper log collection and review is part of every major security standard in existence. As an example, PCI DSS requirement 10 addresses various aspects of logging. Section 10.6 states “Review logs and security events for all system components to identify anomalies or suspicious activity.” 

Log requirements are also explicitly addressed in the Gramm–Leach–Bliley Act (GLBA), Sarbanes-Olxey (SOX), Health Insurance Portability and Accountability Act (HIPAA), and Federal Information Security Management Act (FISMA). 

It is also clear that proper log collection and analysis is critical to an organization’s protection against breaches. The Verizon 2015 PCI Compliance Report compares that company’s findings from their routine PCI certification customers with those customers from their post-breach forensic business. The difference is astounding, with 91.1 percent compliance with PCI requirement 10 in their routine customers, versus 0 percent in their post-breach customers. The importance of proper log handling is clear. 

While proper log analysis is critical, I don’t want to understate the challenge of doing it well. A single Windows server can easily generate over 5,000 log records a day across all categories. Most of these will be routine, and not of interest from a security perspective. That being said, you do have to sift through all of them to find the ones of interest. Without some automation, you might still be looking through Monday’s log entries on Thursday. 

Unfortunately, the challenge only gets worse, because of the number of ancillary systems generating log entries that may have relevance to security. These include: 

  • Access points
  • Firewalls
  • Authentication systems
  • Intrusion detection/protection systems
  • Anti-malware software
  • Application software 

And the list goes on. 

In order for log records to be of forensic value in an investigation, or to be admissible in court, there are more hoops to jump through. Controls must be established to ensure that log records cannot be deleted or altered. Log entries must be monitored to confirm appropriate log access. 

NIST publication 800-92, entitled Guide to Computer Security Log Management, does a good job describing the challenges: 

  • Many log sources
  • Inconsistent log content
  • Differing systems time stamps
  • Inconsistent log formats 

At this point, the complexity of the problem may have you wondering why you should even bother trying to keep up. You could just take the approach that many companies I have worked with do, and ignore them until something goes wrong. 

Unfortunately, by the time you know there is a problem and begin investigating, it may already be too late. Your data may already be for sale on the dark web. My point here is that logs often contain evidence that your network and systems are being probed, long before intrusions actually occur. If you see the warning signs, you have a chance to shut the attacks down before they succeed, but only if you are very proactive. 

I managed security for a SaaS document management company a few years ago. I frequently reviewed system logs from our Web server farm, and was surprised at the number of attempts made by people in other countries, China most frequently, to penetrate their systems using known vulnerabilities. As a result of these reviews, I was able to lock out large blocks of IP addresses, eliminating the problem before it happened. 

So, how do you start an effective log management program? 

Make it a priority

Such a program will cost time and money. Your organization must decide this is important, and allocate appropriate resources to make it work. 

Synchronize your time stamps

Reviewing an incident often involves looking at logs from multiple systems. If you have ever tried to do this using systems that had differing timestamps, you recognize the impossibility of success.

Further, without proper timestamps, your logs would not be admissible as evidence in most courts. You need to use a consistent time source for all of your systems. It is recommended that you use an established outside time source, such as one from this list provided by NIST for a few internal systems, and synchronize the balance of your systems to these. This time synchronization must be done securely, as it is a bit of a security exposure in itself, and you must track and control who makes changes to time information. 


The only practical way to address the issue of logs from many systems is to consolidate them.

There are a variety of products that help with this, including some good SaaS-based offerings, as well as some good open-source systems. I discussed these options, with vendor links, in The one-minute security manager. These products will facilitate collection of logs from various systems into a single repository, and will do some of the work to make the formats common 


When you have your records in one place, you need to filter out the routine entries, so you can focus on those of significance. Some of the above products include good filtering capabilities, but it will take some work, along with trial and error, to get it right. 

Know what to look for

Once you have narrowed your log entries down to a subset of interest, you need to be able to interpret them. Google can be your friend on this, as interpretation varies by the type of log. There are tools that can help as well, such as Apache-scalp for Web logs. 

Save them

It is important that you archive and preserve log entries for later analysis if an event occurs. Different standards require different retention periods. 

Bottom line — log management is a pain, but absolutely essential to preventing security breaches.


Robert C. Covington, the "Go To Guy" for small and medium business security and compliance, is the founder and president of Mr. Covington has B.S. in Computer Science from the University of Miami, with over 30 years of experience in the technology sector, much of it at the senior management level. His functional experience includes major technology implementations, small and large-scale telecom implementation and support, and operations management, with emphasis on high-volume, mission critical environments. His expertise includes compliance, risk management, disaster recovery, information security and IT governance.

Mr. Covington began his Atlanta career with Digital Communications Associates (DCA), a large hardware/software manufacturer, in 1984. He worked at DCA for over 10 years, rising to the position of Director of MIS Operations. He managed the operation of a large 24x7 production data center, as well as the company’s product development data center and centralized test lab.

Mr. Covington also served as the Director of Information Technology for Innotrac, which was at the time one of the fastest growing companies in Atlanta, specializing in product fulfillment. Mr. Covington managed the IT function during a period when it grew from 5 employees to 55, and oversaw a complete replacement of the company’s systems, and the implementation of a world-class call center operation in less than 60 days.

Later, Mr. Covington was the Vice President of Information Systems for Teletrack, a national credit bureau, where he was responsible for information systems and operations, managing the replacement of the company’s complete software and database platform, and the addition of a redundant data center. Under Mr. Covington, the systems and related operations achieved SAS 70 Type II status, and received a high audit rating from the Federal Deposit Insurance Corporation and the Office of the Comptroller of the Currency.

Mr. Covington also served as Director of Information Technology at PowerPlan, a software company providing software for asset-intensive industries such as utilities and mining concerns, and integrating with ERP systems including SAP, Oracle Financials, and Lawson. During his tenure, he redesigned PowerPlan's IT infrastructure using a local/cloud hybrid model, implemented IT governance based on ITIT and COBIT, and managed the development of a new corporate headquarters.

Most recently, Mr. Covington, concerned about the growing risks facing small and medium business, and their lack of access to an experienced CIO, formed togoCIO, an organization focused on providing simple and affordable risk management and information security services.

Mr. Covington currently serves on the board of Act Together Ministries, a non-profit organization focused on helping disadvantaged children, and helping to strengthen families. He also leads technical ministries at ChristChurch Presbyterian. In his spare time, he enjoys hiking and biking.

The opinions expressed in this blog are those of Robert C. Covington and do not necessarily represent those of IDG Communications, Inc., its parent, subsidiary or affiliated companies.

More from this author