Cloudflare: a regex exhausts the CPU

01 · CONTEXT

A new WAF rule, shipped worldwide

In its official post-mortem, Cloudflare says it deployed a new rule in the WAF Managed Rules, the web application firewall that inspects HTTP requests. The rule contained a regular expression that caused enormous backtracking and exhausted the CPU used to serve HTTP and HTTPS. The effect was that the CPU of the network's servers neared its limit everywhere, and customers (and their customers) saw a 502 error page when visiting any of the company's domains.

1. In normal operation, each request reaches an edge server and goes through the WAF before moving on.

2. With the new rule, the WAF starts consuming too much CPU on each edge server, and requests end in errors.

02 · THE REGEX

Backtracking: when matching text costs too much time

The post describes the cause as a poorly written regular expression that created excessive backtracking. The critical part cited is a fragment that greedily combines zero or more characters followed by another zero or more characters and an equals sign. Greedy means matching as much as possible. This is a classic example of a performance antipattern: the cost of a seemingly simple operation depends on the shape of the input.

1. Each request is compared against the rules, including the new regular expression.

2. The regex engine tries many combinations before deciding, and the work falls on the server's CPU.

3. The CPU becomes saturated and there is no capacity left to serve legitimate traffic.

03 · RELEASE

A change path that skipped staged rollout

The bad regex only became a global incident because of how it was released. The standard procedure for a rule change allowed, according to the post, for it to be pushed globally, and let a non-emergency change reach production worldwide without staged rollout. The WAF does not use the staged process by design, due to the need to respond quickly to threats. In addition, the test suite had no way to identify excessive CPU consumption, and the rule passed through it with no sign of a problem. What was missing here was a very common availability patterns practice: limiting the blast radius of a change before it reaches the whole fleet.

1. The engineer publishes the rule and the distribution mechanism sends it to the entire server fleet at once.

2. With no intermediate stage to observe the effect, all servers receive the rule at the same time.

04 · RECOVERY

Turning the component off instead of fixing it

According to the post, the change was made at 13:42, and the first PagerDuty alert indicating a WAF failure went out three minutes later. In the first moments there was speculation about an attack of a kind never seen, which shows how a saturated-CPU symptom can be mistaken for external aggression. The way out was an existing mechanism for disabling a single component worldwide: a team member executed the global WAF shutdown at 14:07, and by 14:09 traffic and CPU were normal everywhere. The company sums up the episode as 27 minutes of downtime. This kind of emergency switch is a form of monitoring and observability put into action: the alert points to the suspect component, and the shutdown limits the damage.

1. The alert points to a WAF failure and the team starts investigating, even considering an attack.

2. The team triggers the global switch that disables the WAF across the whole network.

3. With the WAF no longer consuming CPU, the fleet returns to expected behavior.

05 · LESSONS

What to take to your project

Among the remediation actions listed in the post are reintroducing the protection against excessive CPU use that had been removed, including performance profiling of all rules in the test suite, and changing the procedure so rules go through staged rollout, like the rest of the company's software. From the angle of chaos engineering, it is worth asking which failures of your own release path you have rehearsed.