Don’t fall off the log!

Effective log management is key to data security in the public cloud.

public cloud

In the world of datacenter security, a firewall is often seen as the ultimate protector—the final perimeter of defense. In the public cloud however, the perimeter is fluid, and simple software configuration changes can change the perimeter. You can ruggedize your firewall as much as possible, but it can be bypassed. IT and security managers today are well aware of this possibility and, as a result, use extra layers of security: configuration management solutions, anti-viruses (or other endpoint solutions), WAF, DLP, IDS and other “homemade” monitoring solutions.

The common denominator of all these solutions is the vast array of generated logs because they are needed to thoroughly monitor an IT environment. Log absence results in blind spots in your infrastructure and potentially leaves you vulnerable to threats. Orchestrating a cybersecurity solution typically results in a large volume of logs from multiple sources and systems (typically gigabytes per day). Log management at scale is now a core discipline of security operations in the cloud. Effective log management needs to address two key questions: 1. Which logs to store and how, and 2. How to analyze log data effectively.

When saving logs, you need to carefully consider a few key parameters:

Source variety

Pick and choose various sources of logs that complement each other. For example, network traffic logs along with configuration and settings logs, which will give two different points of view into your environment. Having data from a variety of sources might come in handy in case a hacker who knows her way around penetrates your system. A real pro deletes logs!

Retention time

Typically, IT managers don’t save logs for more than three months, which is perfectly rational since it costs them money and they don’t see the value. However, since the average breach is discovered only after 200 days, saving logs for a shorter period of time is meaningless. If you discover the breach, you’ll have an incredibly hard time doing forensics.

User friendly interface

Saving logs just for the sake of saving them or with an ambiguous plan to use them in the future is time consuming and returns low value. IT managers need an intuitive and accessible interface that includes the functionality that best suits their needs.

Backend systems

It doesn’t matter if you have a homegrown tool or use a 3rd party solution. Make sure the system’s capabilities give you what you currently need and can scale along with your future needs. Perhaps today you’re a single cloud vendor shop, but in the future, you intend to have assets across multiple vendors. You might also want to add external data feeds that come in different formats. Don’t limit yourself to a specific vendor or a specific type of data. The backend should support the volumes, the formats and the correlations and calculations you’re going to use them to perform.

We will look at log analysis and how to make the most out of it in detail in a future post, but for now, let me give you just a few examples of what you can do with your logs:

  • Proof of action. This is extremely useful for compliance and governance. Ever wonder if your IT manager created the best, most secure architecture? You can now query and see for yourself if data is going the wrong way—either to the wrong destination or using the wrong pathways. The necessity of this capability when doing IR and forensics is self-explanatory.
  • Context. A single log might not give you much information about what happened, why it happened and whether it’s OK that it happened. But if you add to the log other pieces of data that will give it context and make more sense of it, the work of security will look totally different and would make more sense.
  • Patterns, anomalies and trends. Large volumes of logs are impossible to use, but these volumes and variety are a haven for big data analysis. Even the sky is not the limit.

However, it’s not enough to have lots of logs and a strong infrastructure to support them. If you don’t have the right skilled people to work with the logs, they are meaningless. Going over alerts, querying them and solving them requires knowledge that comes both from the IT world and the security world. This is a full-time job—not something for your DevOps engineer or CISO to do in their spare time. Here are just a few problems that the lack of right personnel will cause for you:

  • Alerts will either not be handled or not be handled within a proper time frame.
  • Alerts and thresholds will not be fine-tuned, resulting in lots of duplicates and false-positives.
  • The “brilliant” ideas many professionals use to handle the heavy workloads is simply to filter out some of the logs, so they have less to analyze.

If you’re not using logs to enhance your security because you think your cloud provider is doing that for you, you’re wrong. Your cloud provider can give you access to many types of logs and might even have a managed service that alerts you about certain findings, but it will not have the context of your environment and won’t take the follow up actions to better understand the finding and its severity. These are all up to you, and they require the parameters mentioned before.

This article is published as part of the IDG Contributor Network. Want to Join?

SUBSCRIBE! Get the best of CSO delivered to your email inbox.