A few days ago, Cloudflare — along with the rest of the world — learned of a "practical" cache poisoning attack. In this post I’ll walk through the attack and explain how Cloudflare mitigated it for our customers. While any web cache is vulnerable to this attack, Cloudflare is uniquely able to take proactive steps to defend millions of customers.
In addition to the steps we’ve taken, we strongly recommend that customers update their origin web servers to mitigate vulnerabilities. Some popular vendors have applied patches that can be installed right away, including Drupal, Symfony, and Zend.
How a shared web cache works
Say a user requests a cacheable file, index.html. We first check if it’s in cache, and if it’s not, we fetch it from the origin and store it. Subsequent users can request that file from our cache until it expires or gets evicted.
Although contents of a response can vary slightly between requests, customers may want to cache a single version of the file to improve performance:
(See this support page for more info about how to cache HTML with Cloudflare.)
How do we know it’s the same file? We create something called a “cache key” which contains several fields, for example:
- HTTP Scheme
- HTTP Host
- Path
- Query string
- …
In general, if the URL matches, and our customer has told us that a file is cacheable, we will serve the cached file to subsequent users.
How a cache poisoning attack works
In a cache poisoning attack, a malicious user crafts an HTTP request that tricks the origin into producing a “poisoned” version of index.html with the same cache key as an innocuous request. This file may get cached and served to other users:
We take this vulnerability very seriously, because an attacker with no privileges may be able to inject arbitrary data or resources into customer websites.
So how do you trick an origin into producing unexpected output? It turns out that some origins send back data back from HTTP headers that are not part of the cache key.
To give one example, we might observe origin behavior like:
Because this data is returned, unescaped, from the origin, it can be used in scary ways:
Game over — the attacker can now get arbitrary JavaScript to execute on this webpage.
Notifying customers who are at risk
As soon as we learned about this new vulnerability, we wanted to see if any of our customers were vulnerable. We scanned all of our enterprise customer websites and checked if they echoed risky data. We immediately notified these customers about the vulnerability and advised them to update their origin.
Blocking the worst offenders
The next step was to block all requests that contain obviously malicious content — like JavaScript — in an HTTP header. Examples of this include a header with suspicious characters like < or >.
We were able to deploy these changes immediately for all customers who use our WAF. But we weren’t done yet.
A more subtle attack
There are other versions of the attack that could trick a client into downloading an unwanted but innocuous-looking resource, with harmful consequences.
Many requests that have traveled through another proxy before reaching Cloudflare contain the X-Forwarded-Host header. Some origins may rely on this value to serve web pages. For example:
In this case, there’s no way to just block requests with this X-Forwarded-Host header, because it may have a valid purpose. However, we need to ensure that we don’t return this content to any users who didn’t request it!
There are a few ways we could defend against this type of attack. An obvious first answer is to just disable cache. This isn’t a great solution, though, as disabling cache would result in a tremendous amount of traffic on customer origin servers, which defeats the purpose of using Cloudflare.
Another option is to always include every HTTP header and its value in the cache key. However, there are many headers, and many different innocuous values (e.g. User-Agent). If we always included them in our default cache key, performance would degrade, because different users asking for the same content would get different copies, when they could all be effectively served with one.
Solution: include “interesting” header values in the cache key
Instead, we decided to change our cache keys for a request only if we think it may influence the origin response. Our default cache key got a bunch of new values:
- HTTP Scheme
- HTTP Host
- Path
- Query string
- X-Forwarded-Host header
- X-Host header
- X-Forwarded-Scheme header
- …
In order to prevent unnecessary cache sharding, we only include these header values when they differ from what’s in the URL or Host header. For example, if the HTTP Host is www.example.com, and X-Forwarded-Host is also www.example.com, we will not add the X-Forwarded-Host header to the cache key. Of course, it’s still crucial that applications do not send back data from any other headers!
One side effect of this change is that customers who use these headers, and also rely on Purge by URL, may need to specify more headers in their Purge API calls. You can read more detail in this support page.
Conclusion
Cloudflare is committed to protecting our customers. If you notice anything unusual with your account, or have more questions, please contact Cloudflare Support.