Cloudflare Nov 2025: features file doubles in size and the proxy panics

01 · CONTEXT

A feature file that doubled in size

On November 18, 2025, starting at 11:20 UTC, Cloudflare's network began failing significantly at delivering core traffic. The company's official post-mortem is the source of this case and rules out an attack: the problem was not caused, directly or indirectly, by a cyberattack or malicious activity.

The trigger was a change to the permissions of one of the company's database systems, which caused the database to write multiple entries into a “feature file” used by the Bot Management system. The file doubled in size and was propagated to all the machines in the network. The software that reads it had a size limit below the doubled size, and failed.

The file exists because Bot Management includes a machine learning model that generates a bot score for each request, and the model reads this feature file. It is updated every few minutes and published across the whole network; Cloudflare considers it critical to publish it often and quickly because attackers change tactics fast.

02 · THE TRIGGER

An unfiltered query and a preallocated memory limit

The file is generated by a query against a ClickHouse cluster. This query does not filter by database name, and after the 11:05 change it started returning duplicate columns, coming from the underlying tables.

1. The feature file is generated by a query that runs on a ClickHouse cluster, every five minutes.

2. At 11:05, the permissions change makes access to the underlying tables explicit; the query, with no database filter, starts returning duplicate columns.

3. The generated file ends up with many duplicate feature lines and doubles in size.

4. The larger file is propagated to all the machines that make up the network.

5. The file with more than 200 features hits the module's limit and the system panics.

6. The fix stops the generation and propagation of the bad file, manually inserts a good file into the distribution queue, and forces a restart of the core proxy.

Each proxy module has limits to avoid unbounded memory consumption and to preallocate memory for performance. In Bot Management, the limit was 200 features, well above the roughly 60 in use. The bad file, with more than 200, hit the limit and the system panicked.

The FL2 Rust code that performs this check was the source of an unhandled error: the recorded panic was a call to unwrap on an error value, and this resulted in a 5xx response.

The limit itself is a reasonable memory defense. What deserves discussion is what the module does when it receives an input that exceeds it: reject the new file and continue with the previous one, or let the error bubble up and fail the request.

03 · THE REACH

One module, many dependents

Bot Management is one of the core proxy's modules, and it was the source of the outage. The core proxy returned HTTP 5xx errors for all traffic that depended on the bots module.

1. Every request ends up in the HTTP and TLS layers and then enters the core proxy.

2. Workers KV and Access depend on the core proxy and were affected.

3. The Dashboard was affected by the internal use of Workers KV and by Turnstile at login.

4. A patch to Workers KV and bypasses in Access made both go back to an earlier version of the core proxy, where the problem existed with lower impact.

5. With the file already fixed, a backlog of logins and new attempts raised the dashboard's latency.

Besides the 5xx errors, CDN latency increased significantly. The source attributes this to heavy CPU use by debugging systems and observabilidade, which automatically enrich unhandled errors with extra information.

04 · DIAGNOSIS

Fluctuation, suspected attack, and the recovery

The file was generated every five minutes by a query on a ClickHouse cluster that was being updated gradually. Bad data only came out when the query ran on a part that had already been updated, so on each cycle there was a chance of producing either a good or a bad set of files, propagated quickly.

The fluctuation made it confusing what was happening, because the system would recover and then fail again. The team initially suspected, wrongly, a hyperscale DDoS attack. The status page also went down, by coincidence, and this reinforced the suspicion for part of the team.

The initial investigation started from the symptom of degraded responses in Workers KV, and mitigations such as traffic manipulation and account limiting were tried on that service.

The errors continued until the cause was identified and resolved, starting at 14:30. The long tail was the team restarting services that had entered a bad state, with the 5xx volume returning to normal at 17:06.

05 · LESSONS

What Cloudflare set out to change

Among the remediation steps, Cloudflare listed: hardening the ingestion of configuration files it generates itself, as it would with user input; enabling more global kill switches for features; eliminating the possibility of core dumps or other error reports overloading system resources; and reviewing the failure modes of error conditions in all the core proxy's modules.

A file generated by another internal component is still input for whoever consumes it. Validating format, size, and number of items before applying, and having a good previous version at hand, are design questions about disponibilidade.

Discussion question: what changes in the design if a file that needs to reach the whole network quickly goes through a validation step and a staged launch, in the style of the cloud patterns for gradual deployment? What is gained and what is lost in speed?