Security vs. visibility: Why TLS 1.3 has data center admins worried

What's good for the internet may be bad for the enterprise.

fog visibility island

The TLS protocol, which provides the encryption that makes secure web transactions possible, is in dire need of an upgrade. It last saw a big update when version 1.2 was released nearly ten years ago, and the Internet Engineering Task Force (IETF) has for some time been shepherding a major revision of the standard to fruition, which aims to offer improved security by, among other things, jettisoning support for legacy encryption systems.

There's only one snag: a number of data center administrators from large financial, health care and retail corporations have begun to regard the current draft of the 1.3 version of the protocol with increasing alarm. One of the key exchange mechanisms bounced from the draft standard, static RSA, is a crucial tool for admins who want to monitor and troubleshoot data traffic within a company's network. "I think there may be enterprises that don’t realize that this is going to hit them," says Nalini Elkins, President of the Enterprise Data Center Operators (EDCO) consortium. "They’re going to upgrade and things are going to go blind.  They’re going to have outages that they can’t fix and security tools that go dark."

While attempts on a fix are underway, it's worth taking a look at how the community got to this point. It's a story of clashing cultures, differing priorities, and the sometimes convoluted paths by which technical standards make their way to production.

The comforts (and problems) of the status quo

At the core of the TLS protocol is a series of exchanges of cryptographic keys between communicating computer systems, which allow them to talk to each other securely. Static RSA key exchange is one of these, and it's been much loved by data center admins for the visibility it offers into their networks. As Elkins puts it, "In today’s environment, the traffic coming in through the internet is encrypted, and it’s been NAT'd by the content delivery network so we have no way to even find a failing session. We need to get a user name, the URL where he's trying to go, and the time that the failure happened. And we can see that in a packet if we can decrypt it. But if we can’t decrypt the packet, we’re completely blind for troubleshooting."

The advantage of static RSA key exchange is that it allows for that kind of decryption, and for it to take place out-of-band — that is, the packets can be decrypted and inspected by tools that aren't in the main flow of network traffic, which means the various inspection tools can do their work without adding latency that can grind the system to a halt. Plus, these types of tools go beyond troubleshooting to include customer experience monitoring and intrusion and malware detection as well — packets can be traced and decrypted anywhere in the data center.

So why get rid of static RSA key exchange? Well, the problem is that much of what makes it so useful also makes it insecure. The capabilities for out-of-band decryption, which admins have come to rely on for network monitoring, can be abused to snoop on packets on the open internet. There are also potential vulnerabilities to the RSA key mechanism, including ROBOT and key theft. However, if private keys are used inside a data center and not on the internet, then a malicious hacker would have to penetrate that data center and steal both the packets and the RSA private keys to exploit those weaknesses.

Of course, the "I" in IETF means that the internet is the standards body's primary concern. Elkins says that, from the IETF's perspective, the effort to update TLS "has the internet in mind. On the internet, packets transit different equipment, and their intent is to secure that transit. But the point that we’re trying to make is that the enterprise use case is different."

To represent that enterprise use case, EDCO was formed in August of 2017. The consortium consists of participants from manufacturing, insurance, government, retail, finance and other sectors. As Elkins describes it, she read the 112-page TLS 1.3 draft spec and said, "What? They’re taking away RSA? Uh-oh."

"A lot of people assume the vendors are representing us," she said. "We shouldn't stake our future on whether vendors are understanding and communicating issues to the right person.  We need to be involved ourselves! A number of vendors have actually joined EDCO."

The standards model

To try to get the perspective of the IETF on this, CSOonline spoke with the co-chairs of the TLS Working Group, Sean Turner of sn3rd and Joseph Salowey, Senior Product Manager at Tableau Software. Turner and Salowey emphasized that their IETF role, which is separate from their day jobs, is to serve as neutral arbiters and help hash out community consensus on the emerging standard. (They offered their insight and some opinions on the controversy, but made it clear that in doing so that they were speaking for themselves, not for the IETF or working group.)

One thing they agreed on with the folks from EDCO: many working group contributors have a perspective on out-of-band decryption informed by thinking primarily in internet terms.  "If we look at the history of IETF," says Salowey, "part of the community has been concerned about the ability or attempts to introduce things that could be used by a nefarious organization to wiretap or intercept traffic. That segment was concerned about building in monitoring capabilities that some nation-state could use to suppress the people.  The data center is not that scenario, but there were concerns that whatever capability you might build for a data center would not stay in the data center."

Turner sees what he called the "tussle" of security versus visibility here as part of a battle that's been around almost as long as the web itself. "There are other protocols being developed, and some people say, 'We should encrypt this thing.' And then other people are like, 'How are we going to manage that?' From where I’m sitting as the working group chair, I’m just trying to let the conversation happen and then we’re going to judge consensus."

Culture clash

But navigating to a consenus is turning out to be tricky for the EDCO members, not least because the culture of open standards bodies like the IETF is very different from the corporate world. Start with the participants. In theory, anybody can participate in any working group just by participating in the email list. "In terms of IETF process, everything gets 'taken to a list,'" says Salowey.

In practice, however, this makes for self-selecting groups. "Primarily, the group has been made up of a lot of cryptographers, who need to be there, and we’re glad they are," says Elkins. "A large portion of them work for browser developers, and there's some folks from academia, and they’ve done a lot of good work. But their focus has been around security, and they have a strong privacy bent. There needs to be an equal focus on supportability. Otherwise we will end up in situations where the network is very secure and private, but down or slow because we can’t see the packets to figure out the problem." 

"A lot of the discussion is at the code level," adds Elkins. "The average enterprise operational support guy is in a very different world. We don't understand their code-level world completely and they don’t understand our operational world completely."

And some of that failure to understand one another has taken the form of pushback on EDCO's concerns that can feel condescending — in essence, telling them that they should simply rearchitect their networks to avoid the need for out-of-band decryption. "Some people have said, 'You can just transition and stop using TLS and use IPsec,'" says Elkins. "That’s a multi-million dollar project and three or five years. Or they'll say 'Your topology is wrong.' Well, there’s a reason network topologies were built in certain ways. Let’s start with the assumption that the other party is intelligent and understands how to do their job."

Late to the game

But some of the pushback they've gotten comes from the sense that they're intervening late in the process of establishing the TLS 1.3 standard. "If you look at one of the standards that RSA published in 2003," says Turner, "it basically says, 'don’t use this for new applications.' So, it makes it somewhat hard to put out a new version of a protocol that includes it." And a quick look through the working group mailing list archives shows that discussion on removing support for static RSA key exchange from the standard came up as long ago as 2013.

"The conversation about visibility hasn't been happening," admits Elkins. "And when we tried to start that conversation, they said, well, where have you been for the last three or four years? And we have to fall on our swords: we weren’t there. And apparently nobody else was, either. But a data center encrypted with TLS 1.3 isn't supportable. It's a different use case than internet traffic and must be addressed differently."

A fix in process?

"We’re not asking for RSA to come back," says Elkins. Instead, EDCO has proposed a fix. At the IETF 99 conference in Prague last July, the EDCO extended team presented a way to create a static Diffie-Hellman key that would let data center operators continue to inspect traffic on the network in a way similar to how static RSA works today. 

But the rollout didn't go as smoothly as they hoped. Some IETF participants objected because in the proposed fix, the client may not know that traffic inspection is happening, since it isn't obvious on the wire. "It was very contentious," says Elkins. "At the end, they gave us some suggestions on ways to make our solution more privacy-oriented."

The objections became the basis for a new RHRD draft, which allows the client to indicate its agreement to having its traffic inspected. The server cannot perform traffic inspection unless the client agrees; after that agreement, appropriate protocol extensions are created to allow the keys to be exported and reused by the decryption device. The draft will be presented in London at the next IETF conference on March 18-23.

Some bigger picture worries arose in Prague, though, according to Elkins. "They took a hum" — a technique the IETF uses to assess consensus, which involves the participants literally humming in response to proposals — "not on our solution, but on the need for visibility. And the outcome was a 50/50 hum — showing the growing support and need for internal visibility. The risk is that if we don’t gain a supportive 'rough consensus' hum on the proposed solution we plan to present in March, then TLS 1.3 will lose internal visibility. That in turn will cause diminished network manageability and likely cause network performance to suffer until something is done to rectify it. Really understanding and accurately forecasting the impact of future standards is tough but we all want to get it right!"

Salowey was more sanguine about the prospects for resolution. "There was a lot of active discussion, and we weren't able to come to consensus on this being the right approach at that time," he says. "Since then, there's been ongoing discussion. Our main focus within the working group as chairs is to keep the group focused on delivering TLS 1.3 and making sure that product the consensus of the working group. But this is still a topic of discussion within the IETF, beyond just the TLS working group."

Where's it going?

The EDCO members are eager for a resolution. "Troubleshooters can’t migrate to TLS 1.3 the way it is," says Elkins. "It would produce outages of applications that we cannot tolerate. Our ability to troubleshoot problems in a timely manner would deteriorate rapidly to a point where it’s not viable, so we have to come up with a solution in whatever window of time we have."

Salowey and Turner take a more measured view. "This is a new version of a protocol with new characteristics," says Turner. "It takes time for deployment and for other tools to be built up the protocol."

"The reality is," he continues, "additional documents related to TLS 1.3 can and will be specified after TLS 1.3 is published. In other words, there’s going to be an RFC that’s TLS 1.3, which I refer to as the base spec. There will also be other documents that extend TLS 1.3 in the future; these are done in separate documents and are not what I would consider part of the base. (Both of the EDCO proposals to date have been proposed as separate drafts, which is why I’ve said they’re not part of the base spec; it’s not meant to be a diss.) Some of these documents will become RFCs, some will not; some won’t even be processed through the IETF.  For example, an organization or consortia could profile TLS 1.3 to choose which of the options they want to use for their own community."

And work continues. The new version of EDCO's proposal will be presented at IETF 101 in London, which starts on March 18.  "My guess is that the room will be full," says Turner. "And my hope is that we get to the point where we’re discussing the technical aspects of what’s being provided — that [EDCO] makes an ask. Because to this point, the proponents haven’t really asked flat out, 'Are you going to adopt this?' It's been, 'Here’s our problem, then 'Here’s one of our solutions,' and they got a bunch of feedback. I’m hoping in March that’s where we’re going to get to the point where pull the bandage off — are we going to do this or not?"

Meanwhile, EDCO is having its own seminar in London, two days in advance of the conference but not affiliated with the IETF, to discuss the issue and other related points. Elkins put in a pitch for wide participation in the London meeting. "Protocol users don’t usually strive to attend a standards conference," she says. "The developers will say, 'Well, if you care, then come and work with us.' But many of the users don’t know to participate — perhaps wrongly thinking that someone else is looking after their interests. But at the end of the day, they operate based on the available standards, so they should become part of the standards creation process. This allows them to help shape their own destiny and bring added perspective to be considered."

Elkins hopes that the momentum will get more people from industry interested in working on similar issues. "I heard from some people I talked to at other standards bodies: there's not enough industry participation," she says. "And that’s why we formed EDCO. We really need to be at all the standards bodies — ICANN, IETF, ETSI and so on."

No matter what happens in London, it looks like an industry shift is in the works.

Copyright © 2018 IDG Communications, Inc.

7 hot cybersecurity trends (and 2 going cold)