Log management basics

View log management systems as a source of business intelligence—and choose one that fits your business needs. David Torre provides expert guidance.

By David Torre

October 18, 2010 — CSO —

System logs generated by servers and other various network apparatus can create data is in vast quantities, and sooner or later, attempts at managing such information in an off-the-cuff fashion is no longer viable.

Consequently, information systems managers are tasked with devising strategies for taming these volumes of log data to remain compliant with company IT policy, and also to gain holistic visibility across all IT systems deployed throughout the organization. With a tad of guidance and a bit of planning, the recipe for log management is actually straightforward, and the rewards are surprisingly favorable.

What is log management?

First and foremost, a definition of log management is in order. The National Institute for Standards and Technology (NIST) defines log management in Special Publication SP800-92 as: "the process for generating, transmitting, storing, analyzing, and disposing of computer security log data." As you probably knew that much already, what does log management really entail? Put simply, log management is defining what you need to log, how to log it, and how long to retain the information. This ultimately translates into requirements for hardware, software, and of course, policies.

Benefits of log management systems are abundant, and their return on investment is significant. To quantify the value of an investment in this area, it helps to view log management systems as business intelligence systems. Our business is of course information security, but many of the same features and benefits found in traditional BI systems are also present in log management systems. From data extraction, transforming, and loading (ETL) to even back-end enterprise data warehouses, all of the standard BI moving parts are also found in many log management systems.

Also see 'Evaluation criteria for SIEM systems'

The log management system may be a highly specialized business intelligence system in disguise, yet like its business-focused cousin, it brings game-changing benefits to the table. For example, day-to-day transactional data can finally be viewed across the organization as a whole rather than in discrete and disjointed silos. This ability to watch all systems simultaneously is a bit like being everywhere at once. As godly as it may sound, the reality is that this new set of virtual eyes increases your effectiveness without increasing your headcount. Amplified visibility into enterprise-wide events also equates to an increased awareness of real-time activity, which ultimately improves overall security posture by empowering staff to react quickly to malicious events.

Security professionals have long understood the benefits log management systems provide through the centralized storage of logs. Given that it's practically standard operating procedure for hackers to obfuscate their method of intrusion by destroying logs and disabling accounting mechanisms, having a protected and centralized copy of such data ensures that valuable information is preserved for post-mortem analysis, and that evidence is available for any follow-up legal action.

• Print

• Avoiding Pitfalls in Log Management Planning and Selection
• Marcus Ranum on Network Security
• Network Surveillance: Watching the Wires
• Another Look at Log Files