Your cache is exposed

It didn’t have to be this way. This was never meant to be on the Internet.

On February 28, Akamai recorded a 1.35 Tbps DDoS attack against one of our customers. The attack was driven by a relatively new vector, memcached reflection. Possibly the largest publicly disclosed DDoS attack to date, the memcached attack was more than twice the size of the largest DDoS attacks that were launched against an Akamai customer by the operators of the Mirai botnet in September of 2016.

Memcached is a service that is meant to cache data and reduce the strain caused by storage intensive tasks/services. In theory, it is only supposed to be used on an internal/private network to assist with short term storage and delivery of data between systems as part of an application/service. As an example, it finds extensive real-world use as a caching system for production databases, where the application server will check for cached results before doing more expensive database (and possibly filesystem) operations.

Exposure to the outside world

The problem began when developers, seeking higher performance, decided to add UDP networking support to the memcached project. This helps to reduce networking overhead, speeding up the content delivery process and overall performance of the service. Because memcached was only supposed to run on internal networks where trusted machines could communicate with it, there is no authentication mechanism for the service.

Exposing memcached on the public internet before the inclusion of UDP into the software already presented enough privacy and security concerns to make it unthinkable. The addition of UDP simply added ‘It can be used as a DDoS reflector’ to the list of reasons why the service should never be accessible over the Internet.

The primary issue, as it relates to DDoS, isn’t memcached specific, it’s an issue with almost every software/service that uses UDP. With TCP data flows there is a three-way handshake that needs to be completed before real communication can begin. UDP doesn’t require this handshake, which allows spoofed requests and in turn ‘reflection’ attacks. Due to this lack of handshake and other mechanisms present in TCP connections, UDP greatly reduces the overhead and, in turn, speed of the network operations.  It makes servicing the requests faster while also reducing resource consumption.

Along with the fact that it’s fast, the default configuration of memcached supports a stored value of up to one megabyte per key. As with other amplification vectors, you send a small packet and get a larger response — but memcached can reflect/amplify substantially more than other vectors. Memcached can have an amplification factor of roughly 500,000 times larger than the initial query, meaning a 203 byte request results in a 100 megabyte response. This massive amplification factor means a seemingly small set of exposed machines can easily produce huge amounts of reflected traffic, far more traffic than is typically seen in these types of attacks. To further complicate matters, defense strategies against volumetric attacks leave a lot to be desired.  If you’re not using a mitigation provider or have good relationships with your upstream transit providers who can help shed traffic before it hits you, you’re going to have a bad time. The bottom line is you’re going to need fat pipes to ride out this storm.

The wrong solution

There has been some rambling around the Internet about a “flush_all” “kill switch” being suggested as a fix, and even deployed as a feature by a mitigation service/gear provider. The “flush_all” command essentially marks all keys on the memcached server as invalid, allowing the server to purge old data and free up memory. In theory, if an attacker is reflecting payloads off of a memcached server using a key-value pair that they’ve setup, sending the “flush_all” command will cause the keys they’re using in the attack traffic to no longer exist from a querying standpoint. In theory, this could reduce the traffic to near zero as the server fails to look up the payload being requested by the attacker.

This “solution” reminds me of the Mark Twain quote: “Never argue with a fool, onlookers may not be able to tell the difference.”

There are several reasons this is worrisome. First, you’re sending commands to someone else’s server. A server that is already being abused by an unauthorized party. Now you’re abusing it too, just in a different way, which lands you in a very grey area legally. You may be flushing data from the server that it legitimately needs to power an application/service, you don’t know what that server is being used for or what collateral damage you may inflict. Second, requests that have already been made will continue to serve as proverbial data firehoses. The “flush_all” will not work because the request happened before the flush, and that data is still being served to the old request. Third, an attacker only needs milliseconds to reset their key, so unless you continuously spam the server with “flush_all” requests, you likely won’t be able to keep up as you’re now engaged in a battle centered around a race condition.

Order restored…mostly

This all seems pretty “gloom & doom”, but there are some positives that we can take away from this:

There will always be people who are going to exploit services for as long as they can, but there are also a lot of people behind the scenes that want to see these issues fixed. Thanks to the coordinated efforts of the community, we’ve seen memcached reflection capabilities degraded significantly and rapidly since the initial attacks. At the start, there were approximately 100,000 memcached servers available on the Internet. Of those, 52,000 also exposed UDP services capable of being used for DDoS. As of this publication, the estimate is that 43,000 of those UDP memcached services have disappeared. That’s not to say they’re actually gone, but they are no longer publicly reachable due to actions taken on the ISP and server providers’ networks. 

While this is great work by all parties involved and very promising for the future of the Internet and collaboration across our industry, we’re far from worry-free regarding memcached reflection, as even a few hundred of these servers should be a concern due to the overwhelming amplification factors they’re capable of producing.

At this point, memcached has likely run its course and the clean-up efforts have gone well. However, because of the long tail of unpatched/unprotected servers and their potential to generate large floods of traffic, memcached as a DDoS concern will likely be with us in some form for the foreseeable future.