3 ways to monitor encrypted network traffic for malicious activity

Ubiquitous encryption of network traffic prevents you from performing deep packet inspection, but you can still detect and prevent a lot of attacks.

Security experts have been screaming at you for years to encrypt all network traffic. They have a point: Making a secure configuration the default configuration is an obviously good idea. Both the standards and products that implement encryption are very mature. There’s no reason not to!

Well, not entirely. Like everything else in engineering, encrypted traffic presents trade-offs. A big one is that it makes network traffic less transparent to your own security people and monitoring systems. How are you supposed to check network traffic for malicious programs and problematic content?

The short answer is that you can’t. “Deep packet inspection” is not an option. The longer answer is that you can inspect traffic at the endpoints where encryption and decryption are performed and that you can learn a lot just from network traffic metadata, the information in the headers that tell the network where the packet came from and is supposed to go to.

According to Cisco, encrypted traffic nearly doubled from 21 percent in 2015 to 40 percent in 2016. The percentage of encrypted internal enterprise traffic is surely growing rapidly, as enterprise products, such as Microsoft Exchange, are increasingly configured by default to encrypt all traffic.

Most of the network analysis to find malicious traffic in a sea of legitimate encrypted traffic is performed by any decent host- or network-based intrusion and detection systems (IDS/IPS). However, it’s good to be able to go beyond what your tools do and understand your own traffic. The following looks at ways you can do that for protocol-level encryption, not application-level encryption, like that supported in Microsoft Office for data files, nor obfuscation techniques like steganography, which a malicious actor might use to sneak data past your prying eyes.

To continue reading this article register now

Get the best of CSO ... delivered. Sign up for our FREE email newsletters!