Incident · Staff
Google Cloud Jun 2025: null pointer replicated globally via Spanner
Real incident. Key points: progressive deployment and overload control.
Short answer
Real incident. Key points: progressive deployment and overload control. Google and Google Cloud APIs are served by management and control planes distributed by region. They ensure that each request is authorized and passes through the policy and quota checks. The core binary of this verification system is called Service Control.
01 · CONTEXT
A check in the path of every request
Google and Google Cloud APIs are served by management and control planes distributed by region. They ensure that each request is authorized and passes through the policy and quota checks. The core binary of this verification system is called Service Control.
Service Control is a regional service and reads quota and policy information from a regional datastore. The metadata of this datastore is replicated almost instantaneously across the world to manage the quota policies of Google Cloud and its customers.
The API request goes through the management plane, which submits it to Service Control's policy and quota checks.
Service Control is regional and reads quota and policy information from the regional datastore.
The datastore metadata is replicated almost instantaneously to the other regions.
02 · THE LATENT CHANGE
A code path nobody exercised
On May 29, 2025, a new feature was added to Service Control for additional quota policy checks.
The change went through region-by-region rollout, but the code path that failed was never exercised in that rollout, because it required a policy change to trigger it.
As a precaution, the change came with a red button to turn off that policy service path. But it lacked adequate error handling and feature flag protection, and, without the error handling, the null pointer took the binary down.
According to the report, feature flags serve to enable the feature gradually, region by region and project by project, starting with internal projects, to catch problems. With flag protection, the problem would have been caught in staging.
| What existed | What the report says | |
|---|---|---|
| Region-by-region rollout | The change went through it | The path that failed was never exercised in that rollout |
| Feature flag | The change was not protected by a flag | With flag protection, the problem would have been caught in staging |
| Error handling | It was not adequate | The null pointer took the binary down |
| Red button | Came along with the change | Served to turn off that policy service path |
03 · JUNE 12
Blank data, replicated to the world
On June 12, 2025, at around 10:45 PDT, a policy change was inserted into the regional Spanner tables that Service Control uses for policies. Given the global nature of quota management, this metadata was replicated globally within seconds. The policy data contained unintended blank fields.
Service Control pulled the blank fields and exercised the code path with the null pointer, which put the binaries into a crash loop. This happened globally, because every regional deployment was affected.
The result was an increase in 503 errors on external API requests for Google Cloud, Google Workspace, and Google Security Operations products.
A policy change is inserted into the regional Spanner tables. The policy data carries unintended blank fields.
Because quota management is global, the metadata is replicated to the other regions within seconds.
In each region, Service Control pulls the blank fields and triggers the code path with the null pointer; the binaries go into a crash loop.
External API requests start receiving more server errors.
04 · RESPONSE AND RECOVERY
The red button and the herd effect
| Milestone described in the report | Prazo |
|---|---|
| SRE team triaging | Within 2 minutes |
| Root cause identified and red button being put into practice | Within 10 minutes |
| Red button ready for deployment | About 25 minutes after the start |
| Red button deployment completed and recovery starting with the smaller regions | Within 40 minutes |
| Full resolution in us-central-1 | Up to about 2h40 |
In some of the larger regions, such as us-central-1, the restart of the Service Control tasks created a herd effect on the infrastructure they depend on, the Spanner table, and overloaded it. Service Control did not have randomized exponential backoff to avoid this.
Full resolution in us-central-1 took up to about 2h40, with task creation throttled and traffic routed to multi-region databases to reduce the load.
The red button turns off the policy-serving path, and recovery starts with the smaller regions.
In some of the larger regions, restarting the tasks creates a herd effect on the Spanner table they depend on, overloading it.
Service Control had no randomized exponential backoff to avoid the herd effect.
Google throttled task creation and routed traffic to multi-region databases to reduce the load.
Right after recovery, all changes to the Service Control stack and manual policy pushes were frozen until the system was fully remediated.
05 · COMMUNICATION AND LESSONS
When monitoring goes down with it
The first report on Cloud Service Health went out about 1 hour after the crashes began, because the Cloud Service Health infrastructure itself was down due to the failure. For some customers, the monitoramento that ran on Google Cloud also failed, leaving them with no signal of the incident.
| Area | Announced remediation |
|---|---|
| Arquitetura | Modularize Service Control so its functionality is isolated and fails open, so it still serves requests if a check fails |
| Global data | Propagate replication incrementally, with enough time to validate and detect problems |
| Changes | Require changes to critical binaries to be protected by a feature flag and to ship disabled by default |
| Error handling | Improve static analysis and testing to handle errors correctly and, if needed, fail open |
| Sobrecarga | Audit the systems to ensure randomized exponential backoff |
| Communication | Keep the monitoring and operational communication infrastructure running even when Google Cloud and the primary monitoring products are down |
Frequently asked questions
What is Google Cloud Jun 2025: null pointer replicated globally via Spanner?
Real incident. Key points: progressive deployment and overload control.
What does the Google Cloud Jun 2025: null pointer replicated globally via Spanner design cover?
It is organized in these sections: A check in the path of every request, A code path nobody exercised, Blank data, replicated to the world, The red button and the herd effect, and When monitoring goes down with it.
How long is Google Cloud Jun 2025: null pointer replicated globally via Spanner and what level is it?
It takes about 15 minutes to read and is rated Staff level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.