Cloudbleed 2017: parser leaks memory through the CDN edge

01 · CONTEXT

An edge that rewrites other people's HTML

On February 18, 2017, Cloudflare received details of a problem from Google: a Project Zero researcher had seen corrupted pages returned by edge servers. The company's public post-mortem is the source for this case.

Na borda of the network, Cloudflare reads and modifies the HTML of responses in transit, for example to rewrite links, hide parts of the page from bots, or obfuscate email addresses. To do this, it needs to parse the HTML and find the elements that change.

The original parser, written with the Ragel tool, served all of these modifications. The company began writing a new one, cf-html, and was gradually migrating features from the old one to it. Both are NGINX modules that parse buffers containing HTML, make the changes, and pass the buffers along.

1. In the edge HTTP process, the two parsers are filter modules: each receives the buffers with HTML, modifies what is needed, and passes them along.

2. The old parser was the original; the new one was written to replace it, and features were gradually migrating from one to the other.

3. SSL termination ran in an isolated NGINX instance, which the bug did not affect.

02 · THE LATENT BUG

A pointer error that slept for years

The root cause was a pointer error: the end of the buffer was checked with an equality test, and the pointer could skip past the end and keep going, which is called a buffer overflow. A greater-than-or-equal check would have caught the jump. That test was generated automatically by Ragel, which indicated that the tool was not being used correctly.

The code written by the team had an error handler without the fhold, which steps the pointer back by one position. That missing step back was what caused the leak. If the page ended with a broken HTML tag, the error occurred at the end of the buffer and the pointer ended up past the end of the document. Adding the fhold to the handler fixes the problem.

The defect had existed for many years in the old parser, but no memory leaked because of the way NGINX's internal buffers were used. When cf-html was introduced, the buffer usage changed subtly and allowed the leak, even though cf-html itself had no problem.

The difference was in what the parser knew about the last buffer. With both parsers present, the last buffer with data arrived marked as final, and the error handling ran.

For memory to leak, the last buffer with data had to end with a malformed script or image tag and be smaller than 4k. In addition, the customer needed to have enabled a feature that combined the two parsers, such as Email Obfuscation. These conditions explain why the overflow occurred so rarely.

03 · THE IMPACT

One customer's memory on another's page

The leaked memory came from an NGINX-based process that handles HTTP and could contain headers, chunks of POST bodies (possibly with passwords), JSON from API calls, URI parameters, cookies, and credentials such as API keys and OAuth tokens.

Because the infrastructure is shared, a request to a vulnerable site could reveal information from another Cloudflare site.

1. Under certain conditions, the edge HTTP process read past the end of a buffer and returned memory containing private data, such as cookies, tokens, and POST bodies.

2. Because it is shared infrastructure, the returned memory could carry information from another Cloudflare site.

3. Customers' SSL private keys did not leak: the SSL process has a heap separate from the process that handles HTTP.

4. Search engines cached some of the leaked memory through their normal crawling processes.

5. The company, together with the search engines, identified the cached URIs and asked for them to be purged before announcing the problem.

Even with customers' SSL keys protected, a private key used in connections between Cloudflare's own machines leaked, and some internal authentication secrets were in memory.

In the worst period, from February 13 to 18, about 1 in every 3,300,000 HTTP requests could leak memory. The company found no evidence of malicious exploitation.

04 · RESPONSE AND PURGE

Turn off per feature, then clean the caches

A cross-functional team formed in San Francisco and London, and the work was handed off between the offices every 12 hours so the team operated 24 hours a day.

Three features used the same HTML parser chain that caused the leak. Every Cloudflare feature has a feature flag, called a global kill, and turning them off meant memory could no longer be returned in HTTP responses.

1. Three features used the same HTML parser chain that caused the leak.

2. The global kill switch for Email Obfuscation was triggered 47 minutes after the company received the details of the problem.

3. The one for Automatic HTTPS Rewrites was triggered 3h05m later. Turning off a feature took a few seconds worldwide.

4. Server-Side Excludes was also vulnerable and had no global switch, since it predated that mechanism. The company created the switch and deployed a patch to the fleet.

5. With the features turned off, memory could no longer be returned in HTTP responses.

The company notes that the industry standard for fixing a bug like this is usually three months, and that it finished globally in under 7 hours, with initial mitigation in 47 minutes.

Because search engines had cached some of the leaked memory, the security team identified the cached URIs and asked for them to be purged before the public announcement, believing it had a duty of care. With the search engines' help, 770 unique URIs across 161 domains were found.

05 · LESSONS

Hunting the rest of the bug and looking at old code

After finding a pointer overflow in the generated code, the team went looking for others. Part of the security team fuzzed the generated code, another team built tests with malformed pages found on the internet, and engineers inspected the code manually.

They also decided to add explicit checks to every pointer access in the generated code and to log the errors. With this registro, each log line indicated a request that could have leaked memory.