Protocol analyzers: Who needs them, and how to choose one

The complexities of a modern corporate network with multiple physical locations, cloud presence, and even serverless applications means the task of monitoring your network activity requires a significant amount of planning and design. A portion of this design effort begins as part of the initial evaluation phase while choosing a protocol analysis suite.

A big part of that selection is determined by the features most critical to your organization, such as the need to function as an early warning system for attack methods old and new using either traditional analytics or machine learning, support for your corporate cloud presence, or even the need to perform forensic analysis in the event of a confirmed attack.

What is a protocol analyzer?

Protocol analyzers are tools that allow IT administrators and security teams to capture network traffic and perform analysis of the captured data to identify problems with network traffic or potential malicious activity. This traffic data can be observed in real time by a technician for troubleshooting purposes, monitored by an alerting tool to identify active network threats, or retained to perform forensic analysis in the case a network breach is discovered.

Modern networks typically limit traffic to the path between the client and the server by design for performance and security reasons, a key feature of a network switch over the network hubs that were prevalent in the early days of computer networks. This is a deal breaker for casual malicious users looking to steal your corporate data, but only a minor technical hurdle for legitimate network analysis since many of these tools integrate with network hardware or software sensors that can be placed strategically throughout the network. In most cases, network administrators can simply enable a switched port analyzer (SPAN) to mirror network packets passing through the switch to a single port.

In general terms, there are two main categories of protocol analyzers. Ad hoc protocol analysis tools are used to troubleshoot or analyze something specific and can be implemented for little or no cost with minimal planning or technical expertise necessary. However, they are best for focused analysis (a specific protocol or host) and are not suitable for monitoring an entire network for an extended period. Enterprise analysis tools are more well suited for monitoring your entire corporate infrastructure around the clock, with options to provide a variety of alerts when your network is threatened.

Ad hoc protocol analysis

Security professionals occasionally need to break out a protocol analyzer to troubleshoot a problem on the network, such as authentication failures or to confirm sufficient encryption. A variety of tools once competed for the attention of security (and network) professionals in the ad hoc protocol analysis arena, but WireShark (formerly Ethereal) has dominated the space to the point that many others (including Microsoft Network Monitor) were shelved.

The two most critical features of an ad hoc analysis tool, and not coincidentally two key features that make WireShark so successful, is that it be both flexible and easy to use. WireShark’s ability to filter packets either during capture or upon analysis using varying levels of complexity make it a capable tool for everyone from first-time users to seasoned professionals. Since it’s open source and available for all major platforms, WireShark has loads of community support. That removes cost as a barrier and provides a wealth of easily accessible training.

A further benefit is that WireShark can ingest and analyze captured traffic from a number of other protocol analysis tools. That makes it easy to review network traffic during a specific time in history (providing the traffic was being captured) without breaking the budget.

Enterprise protocol analysis

Enterprise protocol analysis differs from ad hoc analysis in key ways, most notably scale and duration. Ad hoc analysis typically occurs when you suspect a problem or need to evaluate a specific network segment, service or application. Enterprise analysis suites need to consume as much of your network traffic as possible on a 24/7 basis to identify patterns, detect anomalies, offer insight into performance bottlenecks, and alert you to traffic patterns consistent with known attack methodologies.

Another aspect to consider when deciding on a protocol analysis suite is whether you plan to use the solution solely to monitor network traffic or to also incorporate audit logs from business applications and servers. This decision will impact cost, because it will determine which solution you settle on and likely the number of administrative users who will be using the tool.

Your user base will not only drive licensing costs, but also training requirements and use cases. Businesses that employ a round-the-clock network operations center (NOC) will have different priorities as compared to those who rely on weekly metrics and system alerts for critical threat notifications.

Well-known monitoring solutions such as Nagios and Paessler PRTG provide holistic monitoring of protocol traffic, database health, application availability, server uptime, and other data points. Other solutions like those offered by Riverbed and ExtraHop specialize in network monitoring, in some cases enabling the tool to offer a more comprehensive feature set and a more intuitive UI.

How to evaluate a protocol analysis solution

The first hurdle to monitoring protocol use throughout your network is enabling consumption of network traffic. This hurdle is complex on a corporate network in a single physical location with multiple network segments but gets interesting when you must monitor multiple geographic locations and cloud-based resources.

Most enterprise-level protocol analysis solutions are flexible when it comes to capturing protocol data. They either offer software agents that you can install in key locations on your network or integrate with network devices that route protocol data to your monitoring tool. Consider the effort needed and control you have using the available capture methods.

Also consider the visibility the protocol analysis solution gives you into your business’s cloud presence. Systems that reside in Amazon Web Services (AWS) or Azure may be some of your more critical business services, and excessive network traffic to these resources might not only indicate malicious activity but can also drive up service costs.

Your business needs may drive your decision more than anything else. If your business focuses on financial or customer health records, security is the driving factor. Features like traffic log retention to enable post-breach forensic analysis, anomaly detection to help identify new threats or network problems and alerting to provide an early warning system will be most important.

For service providers or content providers, or even businesses that provide web-based applications, performance metrics may be the focal point.

Any business information system is only as good as the useful information you can get out of it, and protocol analyzers are no exception. Many of the enterprise protocol analysis suites on the market provide built-in dashboards you can customize.

Even if the solution you select offers built-in analysis tools, you might get more value out of cloud-centric analytics systems like Splunk or Datadog, which feature analysis tools that leverage machine learning for anomaly detection and even correlate multiple data types such as protocol analysis and log monitoring. These third-party analysis solutions often give you a more complete view into your data, but also come with additional licensing and training costs of their own.