Avoiding Pitfalls in Log Management Planning and Selection

Over the past decade cyber security has emerged as an important concern for organizations of all sizes. The increase in digitized corporate records, coupled with the rise in cyber crime, is driving organizations in the public and private sectors to invest in more protection for sensitive data and regulated or other critical assets. In just the first two months this year, the Privacy Rights Clearinghouse has noted data breaches at several financial, healthcare and educational institutions as well at federal, state and local governmental agencies.

While private businesses may store specific pieces of information about a consumer—such as a credit card number or a medical record—in different departments, governments process and store enough information to entirely reconstruct an identity. The risk they must address goes well beyond consumer identity theft. Governments conduct research and development in numerous areas, including biotechnology and military advancement. They manage and regulate the transportation and utilities infrastructures. All of these functions rely heavily on information systems which, if compromised, would have a widespread impact and tremendous cost.

Monitoring and Log Management5 Things You Can’t See on Your Network.]

Fundamentally, protecting IT assets in the public or private sector requires visibility into activity occurring on networks. But with so much happening at any given time—employees logging in and out of applications, badge swipes, email communications, opening and closing of sensitive files etc, simply capturing and making sense of network activity in itself a huge challenge. This is where effective log management can make a huge difference. [Editor’s note: See also

Logs provide a minimally intrusive means of gaining visibility into all user, system, and application activities. With proper planning, selection, and deployment of a log management solution, organizations can proactively detect threats, breaches, and policy violations, while also reducing the costs and efforts associated with regulatory compliance. Yet, across the planning and selection phases of log management important criteria and considerations are often overlooked.

Planning Phase

In the planning phase, the most common oversight is inadequate consideration of long term use cases and drivers. Any organization might begin its search for a log management solution with a given driver in mind, such as perimeter device monitoring. Over time, most will expand into broader use cases such as privileged user monitoring or regulatory compliance with FISMA, HIPAA, and PCI, etc. This trend highlights the importance of evaluating the functional breadth and the scalability of any log management solution up front.

A common driver for functional breadth arises as use cases transition from requiring historical analysis (which is integral for regulatory compliance) to robust real time correlation capabilities (for scenarios such as user activity monitoring or sensitive data protection). Solutions that do not offer an integrated growth path from historical to real time analysis (or vice versa) will eventually require a second investment with redundant log collection and storage layers.

Scale in log management has several dimensions. For example, expanding a log management investment from perimeter threat monitoring to regulatory compliance will increase the number and type of assets that need to be monitored significantly. In turn, the total event volume that must be supported also rises. Given the long term retention requirements that accompany regulations, capacity also now becomes a challenge. Depending on how distributed regulated assets are, geographic scalability becomes a must have. Finally, each use case adds additional load in terms of analysis and all these dimensions of log management scalability should be considered as part of the planning process.

Evaluation Phase

When planning is done right, short-listing vendors for evaluation becomes much easier since the test requirements are well defined and aligned with long term goals. However, as part of this process, evaluation of many factors such as vendor independence, viability, evaluation of support and services, and relevant reference accounts is often overlooked.

Across use cases, any organization will need to monitor devices all the way from the physical layer up through custom applications; this infrastructure will rarely come from a single or even a handful of vendors. Yet several log management vendors have very limited out of the box support for a broad range of devices. Larger vendors may offer breadth in collection capabilities, but it is often limited to sources from their own portfolio. When you look across layers of the OS stack, the infrastructure at most organizations will be heterogeneous so support for the entire range of vendor and device logs in the environment (not just the immediate use case) is an important evaluation criterion.

Technology is only one aspect of any IT investment. With the downturn in the economy, many vendors are hard hit financially. Before making any investment, it is important to evaluate the viability of the vendor, independent of their technology. Along the same lines, the quality of support, services and partnerships should be evaluated. Don’t assume that a larger vendor can meet your needs best. A more accurate metric would be the size of the support and services staff dedicated to log management. Otherwise you may end up having to go through three tiers of escalation before actually speaking with a specialist in log management.

Finally, organizations in different verticals may differ in the type of devices they have. References from deployments of equal scale are invaluable in ensuring that solutions under consideration can in fact meet your needs in terms of technology, support, and services.

This isn’t a comprehensive discussion of the common pitfalls in log management, but hopefully it sheds more insight on common mistakes that can be avoided in planning and selecting the right log management investment. ##

Ansh Patnaik is the director of product marketing at ArcSight. He is an ISSA and ISACA member and maintains the CISSP certification. Ansh has worked in the security space for over 10 years with companies such as BindView/Symantec and Omniva Policy Systems.