Bleeding clouds: Cloudflare server errors blamed for leaked customer data

Problem discovered by accident, Cloudflare said to be downplaying the incident

data spill leak
Credit: Thinkstock

While working on something completely unrelated, Google security researcher, Tavis Ormandy, recently discovered that Cloudflare was leaking a wide range of sensitive information, which could have included everything from cookies and tokens, to credentials.

Cloudflare moved quickly to fix things, but their postmortem downplays the risk to customers, Ormandy said.

The problem on Cloudflare's side, which impacted big brands like Uber, Fitbit, 1Password, and OKCupid, was a memory leak. The flaw resulted in the exposure of "HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data," Cloudflare said.

Complicating matters, the leaked data was being cached by search engines.

About an hour after being alerted by Ormandy, Cloudflare disabled three features on its platform; email obfuscation, Server-side Excludes and Automatic HTTPS Rewrites, as they were using the broken HTML parser chain determined to be the cause of the problem.

According to Cloudflare, the problem could have started five months ago, on September 22, 2016.

"The greatest period of impact was from February 13 and February 18 with around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage (that’s about 0.00003% of requests)," a blog post by Cloudflare's CTO, John Graham-Cumming, explains.

In an email exchange, Cloudflare pointed Ormandy to the company bug bounty, which offers a reward of a t-shirt instead of financial compensation, leading Ormandy to speculate the company doesn't take the program seriously. As the disclosure deadline quickly approached, Cloudflare engineers worked around the clock to resolve the problem.

Google has started removing cached copies of the leaked data, but other search engines are still holding some copies.

As an example of how wide-reaching the problem was, and how random the data leak became, we located Fitbit that was pushed to a website in the Philippines.

Server administrators are advised to use their best judgement when it comes to revoking and reissuing certificates, as well as rotating any critical keys or passwords.

While password changes wouldn't hurt for end users concerned about this issue, it's unclear exactly what options are going to be made available to Cloudflare customers, and the users exposed by this incident.

"The examples we're finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I've informed Cloudflare what I'm working on. I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full HTTPS requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.," Ormandy noted in a Project Zero ticket on the incident.

As mentioned, Uber, 1Password, Fitbit, and OKCupid are just some of the known brands affected by the flawed Cloudflare code. There is a running list of impacted domains available on GitHub, last count pegged the total at more than 4 million domains.

However, at the time this story was published, only 1Password has issued a statement on the incident, assuring customers that their passwords were safe.

"No 1Password data is put at any risk through the bug reported about CloudFlare. 1Password does not depend on the secrecy of SSL/TLS for your security. The security of your 1Password data remains safe and solid," the statement explains.

Salted Hash has reached out to several brands for comment, both on Twitter and offline. If any of them respond, we'll update this post.

Update:

John Graham Cumming, Cloudflare CTO, responded to questions earlier this morning.

With regard to contacting customers:

"We are currently involved in ongoing dialogue with our customers and have given them information about the best way to notify us if they have questions."

As to comments that they were downplaying the seriousness of the issue:

"We have written a very detailed blog post recounting all of our experiences identifying, fixing, and neutralizing the impact of a bug that was discovered in our system on Friday, February 17. From the moment we were notified of this bug, an internal team at Cloudflare has been working 24 hours a day to address it.

"We've also been working with all of the major search engine providers to protect customers by removing any sensitive data inadvertently cached. The industry standard time to fix a bug like this is three months. Within 47 minutes of being notified, Cloudflare deployed an initial mitigation. We were completely finished with mitigating the bug in seven hours. We have worked quickly and taken this matter very seriously from the moment we were alerted to it."

On Twitter LastPass told customers that their product was not impacted as they don't use Cloudflare. At the bottom of this support document, a list of websites that support One-Click password changes is available. The list is limited, but many of the websites use Cloudflare, so it is a good idea to rotate passwords in the chance that doing so will reset authorization tokens.

Lots of vendors have written-in with comments and observations since the Cloudflare story started to spread, but one caught our attention:

"... A lot of popular internet companies/operators have been affected - and unfortunately they'll have to be the ones working directly with customers and giving them the bad news. All affected sites/services need to destroy all HTTP sessions and potentially do API key as well as password resets across the board..." - Kunal Anand, CTO and Co-Founder, Prevoty

Add your thoughts to our Facebook page.

To comment on this article and other CSO content, visit our Facebook page or our Twitter stream.
Healthcare records for sale on Dark Web