CrowdStrike 2024: Channel File 291 takes down 8.5 million machines

01 · CONTEXT

A kernel sensor and two update paths

On July 19, 2024, a content update to CrowdStrike's Falcon sensor took down Windows hosts. The source for this case is the root cause analysis (RCA) report published by the company itself, complemented by its preliminary incident report. The sensor reaches the customer through two paths: Sensor Content, shipped along with the sensor, and Rapid Response Content, delivered from the cloud to react quickly to new threats.

Rapid Response Content is configuration, not code. Sensor Content, on the other hand, follows a staged release, with internal use, early adopters, and only then general availability, and the customer chooses which version to install. The RCA is precisely about bringing that staged discipline to the fast path.

1. Detection engineers create new rules, and the configuration system turns them into template instances.

2. Before publication, the validator checks the content against the template type definition.

3. The channel file is published and delivered to online sensors.

4. On the host, the sensor runs as a kernel driver: an error in it takes down the entire system, not just the application.

02 · THE INCIDENT

A short window, an entire Windows fleet

The faulty update was published at 04:09 UTC on July 19, 2024 and reverted at 05:27 UTC. Anyone who was online in that window and received the channel file was affected; anyone who connected after the revert, or did not connect during the window, was not.

On July 29, the company reported that about 99% of Windows sensors were online compared with before the update.

03 · THE ROOT CAUSE

Twenty values delivered, twenty-one expected

The cause was not a single error but the combination of three: the template type definition called for 21 inputs while the integration code supplied 20, the interpreter had a latent out-of-bounds read, and no test exercised a non-wildcard criterion in the 21st field.

1. The template type definition said it received 21 input fields, and the validator checked the content against that definition.

2. The sensor's integration code gave the interpreter only 20 values, one fewer than the definition promised.

3. In the tests and the first instances, the 21st field used a wildcard, so the test did not expose the missing value.

4. On July 19, an instance came in with a non-wildcard criterion in the 21st field. The validator, which assumed 21 inputs, approved it.

5. The interpreter tried to read the 21st value, which did not exist: an out-of-bounds read, an unhandled exception, and a system crash.

Memory dump analysis points to the csagent.sys driver, a kernel driver, as the source of the invalid read.

This is a classic case of an inconsistent contract: two parts of the system disagreed on the data format, and each one passed the checks that were based on its own assumption.

04 · WHY NOBODY SAW IT

Passing tests and inherited confidence

The automated tests used a static set of 12 cases, and the test channel included a wildcard criterion in the 21st field of every instance. As a result, the out-of-bounds read was never exercised. Between April 8 and 24, 2024, three additional instances were deployed and worked as expected.

The decision to deploy the July instances weighed the tests done before the template type launched, confidence in the validator's checks, and earlier successful deployments.

05 · WHAT CHANGED

Closing the cause and reducing the blast radius

Besides closing the cause, the RCA addresses limiting the damage of a next failure. The finding is direct: each template instance must be deployed in stages. Instances approved in the canary are promoted to progressively wider rings, or reverted if problems arise, and promotion to the next ring comes with extra observation time, during which telemetry is collected.

1. Validated content goes first to a canary ring, a small group of sensors.

2. Canary telemetry shows whether there are system crashes, a rise in false detections, or performance problems.

3. With healthy signals, the system promotes the content. With bad signals, it reverts.

4. Promotion proceeds ring by ring until the full fleet, and each ring accumulates observation time before the next.

5. The customer also gains control over where and when content is deployed.

06 · LESSONS

What to take into your design

The first lesson is about disponibilidade: every path that reaches the entire fleet must have a stage that limits the damage before the failure signal, whether it is code or configuration. The second is about observabilidade: promoting content only makes sense if the previous stage's telemetry can say it is healthy.

The third is about recovery: reverting the content at the source does not, by itself, fix the machines that already loaded the faulty file. A disaster recovery plan must account for the case where the agent that failed is on all hosts at the same time.