Is the Unlimited PCAP Pipe Dream Now an Affordable Reality?

At my first job as a Firewall Technical Support Engineer, we always said, “Packets Never Lie” – and this mantra has helped me numerous times throughout my career. Full-Fidelity packet capture (PCAP) is the Holy Grail for security analysts and threat hunters. Today, too many organizations rely purely on logs for incident response, and while logs tell you something happened – a connection was made, bytes were transferred – they don’t tell you what happened.

With full packet data an analyst can recreate activities that led up to a security event, understand how it happened, accurately measure the impact to the business and implement changes to prevent similar events in the future. Full packet data is also incredibly valuable because it enables organizations to hunt proactively for unknown threats on their networks, and to know whether new zero-day threats that are found today may have impacted them in the past.

The challenge with PCAP is that retention takes lots of storage and processing it takes lots of computing power. Organizations that have legacy packet capture products typically only store a couple weeks, and this storage is fixed and inelastic. It’s also extremely time-consuming and difficult to correlate and manage threat intelligence with traditional packet capture products. As data volumes grow rapidly on enterprise networks, these environments quickly retain less and less data, and provide diminishing value.

It is not uncommon for an incident responder to attempt to access full packet data related to a particular security event only to be informed that this data is no longer available. These limitations characterize most on-premises products while the cloud enables responders to move beyond hardware costs and capacity limitations. Using the power of the cloud, unlimited packet capture becomes an achievable reality for any organization.

Not just dollars, but sense

Forward-thinking organizations know they need to capture more packets and to keep them longer than just days or weeks. The cloud opens the door to a whole new way of thinking about network security that lets them do just that – and more. The costs of cloud storage are now calculated in fractions of cents, and cloud computing prices have a lower TCO than proprietary data centers.

This new dynamic creates tremendous opportunity for organizations that want to realize the time and cost savings of moving their workloads from on-premises equipment to the cloud. Its flexibility enables organizations to retain data for as long as necessary, which is great for those keen on scaling storage requirements to meet specific business needs.

Rather than storing just raw PCAPs, intelligently designed security solutions allow organizations to fine tune the types of data they capture, and optimize and compress data to help provide greater value to balance costs. With more of the right types of information to work with, analysts gain critical visibility into the activities on their networks, and also hindsight into past activities that traditional, appliance-based PCAP products can’t deliver.

But saving and serving up files is just the tip of the cloud technological iceberg.

No hardware, no problem

Being able to spin up a server or add more storage at the click of a mouse is already a compelling argument for IT departments. After all, they’d much rather move a mouse than purchase and manage expensive servers. The cloud enables organizations to spin up and tear down infrastructure as needed and on-demand. It also enables organizations to pay for what they use versus having to over provision and buy more than they need because they may need it in the future.

Security teams are also following this trend. They realize their operations are improved significantly when they don’t have to consider the cost and burden of deploying and managing appliances required by legacy security products. Having to install and configure racks of equipment, manage upgrades, build databases and manage infrastructure wastes valuable hours from every day that should be spent on securing the organization.

It’s just sound logic: if an organization’s storage and workloads are already in the cloud, security to protect these should be there too.

A stitch in time saves nine

Cost savings aren’t limited to deploying and managing security products in the cloud. These savings are realized in day-to-day operations too. Storing unlimited PCAPs centrally means analysts can be more efficient in their investigations since they don’t have to waste time moving between different systems to collect data. It makes it easier to correlate the data you capture with information from other security products for greater context from the network core to the endpoint.

For threat hunters, that also means they can dive deep and wide into past security events to discover unknown network compromises. If you don’t have a threat hunting team, you should start thinking about building out your threat hunting practice today. All of that adds up to having a faster, more efficient security team that can reduce attack dwell time significantly, or even find threats before they become full-blown attacks.

Search speed daemons

Performing forensic analysis and exploration can be resource-intensive, and extracting information from captured data and performing complex queries on a large body of data can take a very long time. With the cloud’s unlimited computing power, indexing and performing complex queries on data is much faster, making searches far less frustrating for security teams.

That means they can spend more time hunting for threats rather than hunting for data, which is perhaps the greatest benefit of being able to retain and actually use an unlimited history of full PCAPs. It means organizations are better protected because analysts are working with a full set of data. It gives them a much larger breach detection window so they can catch threats earlier in the cyber kill chain, enabling them to prevent data loss.

Affordable reality, unlimited benefits

We’re still in the early days of the cloud, and it’s likely performance, scalability and overall value will continue to expand. We can’t say for certain what the next practical application will be for the cloud, but it’s certain that whatever it is, it’ll need to be kept secure. And security teams must continue to embrace the cloud to improve their detections, focus their resources and to have access to the data they need to actually to do their jobs.