The Heartbleed OpenSSL flaw is worse than you think

The OpenSSL flaw named Heartbleed is pretty huge. Many of us in the computer security industry are prone to hyperbole when a big exploit in a popular piece of software is announced, but I can’t put it any better than Bruce Schneier did when he said, “On the scale of 1 to 10, this is an 11.”

OpenSSL is a very popular open source service implementation that uses the SSL and TLS protocols. It is the backbone for literally tens of thousands of other programs and services that allow SSL or TLS. It’s used in Apache, Nginx, and most open source operating systems (such as Linux and BSD) distributions. OpenSSL probably runs on 60 percent or more of the websites that offer HTTPS connections and is used for many other popular services that use SSL-/TLS-based protocols, like POP/S, IMAP/S, and VPNs.

There’s a very good chance that if you can connect to an SSL-/TLS-based service and it’s not running Microsoft Windows or Apple OS X, it’s vulnerable. This includes most VPN appliances, copy machines, and even most appliances. If you can connect to it using HTTPS, and it’s not running on Microsoft Windows or OS X consider it vulnerable until proven otherwise.

Note: Even computers running Microsoft Windows or OS X (which are not vulnerable by default) could be running software or services that are.

The bug, named Heartbleed, because it has to do with a routine common feature known as a heartbeat, was first introduced in December 2011, and wasn’t removed from the OpenSSL code base until April 7, 2014. If you have anything running OpenSSL versions 1.0.1 to 1.0.1f (inclusive), it’s vulnerable. The best summary site can be found at Heartbleed.com. Here’s a good example of how the bug could be exploited and the potential impact.

When this bug appeared and I confirmed how easy it was, I started calling friends who I knew had tons of OpenSSL. Interesting, most of them felt that they either had it under control or didn’t understand the scope. If you think you’ve patched the few computers you have that use OpenSSL and have called it a day, you probably don’t understand how pervasive OpenSSL is. It is probably the way you connect to your security appliances. It’s probably the way you connect to your routers, gateways, and switches. It’s probably the way you connect to many of your (non-Microsoft) cloud services. To thoroughly mitigate the OpenSSL bug means interrogating your own environment and researching those you connect to.

For OpenSSL under your control

Step 1: Find and inventory OpenSSL instances

Start by looking for all instances of OpenSSL in your environment. A quick way to do that is to conduct a TCP port scan for default SSL and TLS ports using your favorite port scanner, and rule out those that are running Microsoft Windows, OS X, or other non-OpenSSL products. SSH, on TCP port 22 and HTTPS on 443, is a good start, but there are literally dozens of protocols and ports that could be using OpenSSL.

I came up with this quick list: 22 (SSH), 26 (encrypted SMTP), 443 (HTTPS), 563 (NNTP encrypted), 636 (LDAP over SSL), 989 and 990 (FTP encrypted), 992 (Telnet encrypted), 993 (IMAPS), 994 (IRC encrypted), and 995 (POP3S). There are probably dozens of other protocols that could use OpenSSL for its SSL/TLS protection, and any of these services could ultimately be running on any port (if not configured to use the well-known default port). Don’t forget to include all the websites hosted external from your company but owned or managed by your team.

Step 2: Determine if port is running a vulnerable version of OpenSSL

After you’ve located your IP addresses and ports potentially running OpenSSL, run a scanner that can do a test connection and return the answering connection string (which often returns the version). Many scanners, like the very popular and feature-rich Nmap scanner (nmap.org), can do the port scanning and identify the vendor and version number at the same time. Very handy.

Step 3: Prioritize vulnerable assets

Next, identify which assets are vulnerable and what risk they exposure your company to. An Internet-exposed service containing confidential information, logon creds, and other critical services should certainly be sent to the top of the list. I would add security appliances and network devices there as well.

Assets like your copy machine may or may not need immediate remediation. Some readers might think that a copy machine might not never need remediation. But if you connect to your networked company machine (or printer) using a browser over HTTPS, it’s probably running OpenSSL. And network copy machines and printers are often connected using directory service credentials. Some are even accessible over the Internet. Your mileage may vary.

Step 4: Upgrade what you can

If possible, install a nonvulnerable version of OpenSSL. An alternative is to recompile an existing vulnerable version using the compile time option -DOPENSSL_NO_HeartbeatS.

Step 5: Generate a new key pair

There is always the possibility that your OpenSSL private key was compromised by a previously successful Heartbleed attack. If you plan to continue using a nonvulnerable version of OpenSSL, make sure to generate a new private/public key pair, and begin using that from now on.

Step 6: Mitigate the remaining vulnerable versions

There will probably be instances of vulnerable versions of OpenSSL that you cannot upgrade. It could be mission-critical systems that simply can’t be upgraded without going through a huge change control process, or it could be an appliance, device, or software that you literally don’t have access to.

In those instances, it makes great sense to keep the bad guys from being able to exploit your vulnerable versions, using any type of filtering (such ad IP whitelisting, domain whitelisting, firewalls, routers, and VLANs.).

Keeping unauthorized parties away from your OpenSSL connections is a good thing to do after you’ve identified vulnerable versions, but before you’ve remediated them. Actually, it’s a great idea to do all the time with or without a known vulnerability.

For systems not under your control

If you look, you’re likely to find many systems you rely upon, either inside your environment or outside of it, that need remediation. Contact their owners or vendors, let them know you’ve identified a vulnerable version, and ask how they should be remediated.

You’re probably not directly aware of every external website that your company relies on. Consider filtering for outbound (or even inbound) SSL/TLS connections to create another list of potential OpenSSL services. The links I’ve given you above already including information that allows you to test Internet-accessible websites.

Note: The law is not clear on whether “testing” is legal or not. Only test the websites you own or are sure you are legally allowed to test.

Destroy and rebuild

Most serious experts are saying to assume complete compromise on every vulnerable OpenSSL system, and to change every secret associated with the possible compromised system. This means to change all logon credentials stored or access through the system, and contemplate what it might mean if the data on it was compromise, stolen, or adulterated.

If you’re going down this route, you probably want to rebuild the system from scratch. When any compromised system you can never be sure that the underlying OS is not been modified in a way to allow a permanent backdoor.

It’s worse than bad

Sadly, there are likely to be literally millions of vulnerable installs left unpatched and unmitigated in the real world. Sure, we’re going to fix what we find and are allowed to remediate. But so many systems that have a vulnerable version of OpenSSL are probably owned and operated by people that haven’t heard about Heartbleed, and won’t hear about it. Appliances and systems installed by other people, and even devices they bought and installed by have no clue about whether it is using OpenSSL, much less whether it is vulnerable.

How about that wireless router you have installed at home? How about that new fancy TV? Can you connect to it using HTTPS? If you can, it’s probably running an open source Web server and that open source Web server might be running OpenSSL. Perhaps the only saving grace, for millions of people, is that their OpenSSL-enabled device was likely compiled before the last two years, and so they don’t contain the Heartbeat vulnerability. Of course if they haven’t been updated in two years, they are likely running tons of other vulnerabilities, but at least they aren’t all being publicly discussed at the moment.

Do your best due diligence to make sure that you and your company is covered. This isn’t just about external, Internet-facing websites. The bad guys routinely get on the internal networks and you can bet that they will be looking for vulnerable versions of OpenSSL with vigor.