How to track down hacks with log files

Most malicious computer attacks leave telltale evidence in the victim’s security event logs. The Verizon Data Breach Investigation Reports have been bringing word on this for many years.

My favorite quote is from the 2008 DBIR, which says, “In 82 percent of cases … the victim possessed the ability to discover the breach had they been more diligent in monitoring and analyzing event-related information available to them at the time of the incident.”

The glut of evidence collected by computers could be used in an episode of “CSI.” In real-world courtrooms, crime experts remind juries all the time that the elaborate evidence collection shown in “CSI” almost never happens in real life. In the computer world, it’s the reverse: The digital evidence is collected all the time, but no one bothers to look at it.

Of course, it’s more complicated. Most event logs are primarily useless noise that obscures the entries most defenders should be analyzing. I’m a huge proponent of “less is more” event logging, where thoughtful filtering significantly improves the value of logs. When I hear a security event log team buying petabytes of disk storage for their event logs, I know they’re inefficient.

It gets worse. Most companies lack a clear understanding of what logs they have — or should be collecting — not to mention which types of malicious events these logs might possibly detect.

Over the last year I’ve seen many organizations creating matrices (usually in spreadsheets) that detail every log that the company’s assets are either generating or could generate. It includes a list of every computer device (and sometimes written logs to capture physical attacks), as well as workstations, mobile devices, servers, routers, firewalls, proxies, switches, antimalware software, application logs, and more. Many of these devices generate dozens of logs.

Take a Microsoft Windows server, for example. Everyone has worked with the basic Windows event logs — Application, Security, and System — for years. But ever since Windows Vista and Windows Server 2008, Windows event logs have included dozens of filter logs, each detailing a particular application or process. Examples include AppLocker, Authentication, BitLocker, Bluetooth, Code Integrity, Group Policy, NTFS, Task Scheduler, UAC-FileVirtualization, and WER-Diagnostics. The individual filtered logs are a great way to focus your forensics.

Beyond built-in (or custom-created) Windows event logs, a typical Windows computer may have a handful to dozens of other logs. Search on *.log to see what I mean. Web servers have Web server logs. If you’re using Windows Firewall, events are usually saved to a file called pfirewall.log. You’ll find install logs, Windows Update logs, patch logs, diagnostic logs, VPN logs, and usually dozens of application logs. Non-Windows computers will have many, many log files as well. Even your antimalware systems and devices have multiple log files.

How should you deal with this glut of data? Start with these two steps:

1. Do an inventory. List all devices that have log files, the reason for the log file, the names and locations of the log files, log formats, possible events, current and maximum log file sizes, and anything else that might prove useful (rotation method, backup method, retention period, and so on).
2. Determine which type of malicious events your log files can detect. Create a matrix along the other axis of the spreadsheet, then highlight the strengths, gaps, and weaknesses. I often see companies use colored shading (such as red, yellow, and green) to quickly highlight the conclusion.

Even though your event log matrix is likely to be dozens of rows and columns wide, in one place you’ll quickly be able to note your areas of opportunity. You can fine-tune your event log collection process or buy additional tools along the way. Without creating this type of matrix, you’ll never be aware of your strengths and weaknesses — at least, not at a glance.

My favorite part of the process is finding areas of synergy that you never knew you had. For example, I’ve been pushing my customers to enable application control programs to generate events whenever a new, unexpected program or process is executed. Whenever a new malicious program or process is noted (and hopefully stopped) by the antimalware software, the detection can be compared to when the program or event first appeared.

The difference between those two events is the real risk horizon for that malicious event. I call it mean time to detection. The more you can shorten this metric, the better job your antimalware software is doing, and the lower the risk you have (from that particular scenario).

Good event management isn’t easy. But without an accurate event log matrix you won’t have a solid understanding of what you have and what needs to be fixed. Go ahead, take the red pill, and see the real matrix!