Incident · Staff
Google Dec 2020: quota reported as zero takes down global login
Real incident. Key points: authentication and authorization.
Short answer
Real incident. Key points: authentication and authorization. Google maintains a central identity service, the User ID Service, which holds each account's unique identifier and handles OAuth credentials and cookies. Account data lives in a distributed database that coordinates updates with Paxos, and the service rejects requests when it detects stale data.
01 · CONTEXT
Identity as a dependency of everything
Google maintains a central identity service, the User ID Service, which holds each account's unique identifier and handles OAuth credentials and cookies. Account data lives in a distributed database that coordinates updates with Paxos, and the service rejects requests when it detects stale data.
Users and apps call services that require login.
Every authenticated service needs the identity service to verify the credential before acting.
The identity service stores accounts in a distributed database coordinated by Paxos and rejects stale data.
That is why identity is a dependency at the scale of global: Workspace apps depend on the account infrastructure for login, authentication, and access control to resources.
02 · TRIGGER
A quota that believed in zero
Google uses automation tools to manage the quota of resources allocated to each service. The User ID Service was being migrated to a new quota system, and a change made in October registered it in that system, but parts of the previous system stayed in place and reported the service's usage as zero.
There was a grace period in the enforcement of quota restrictions, which delayed the impact. When it expired, the automation reduced the quota allowed to the identity service, and the incident began.
The residual measurement reports to the quota system that the identity service uses no resources at all.
Once the grace period expires, the automation reduces the quota allowed to the identity service.
With the quota reduced, the Paxos leader of the accounts database cannot write.
Soon after, most reads become stale and authentication queries start to fail.
The existing safety checks did not cover the scenario of zero load reported for a single service. The Register's report, which cites the report, says the same about the tests of the time. The table shows why each check let the change through.
| Safety check | Why it did not catch it |
|---|---|
| Quota change for many users | Only one group was the target of the change |
| Quota below usage | Usage was reported, incorrectly, as zero |
| Excessive reduction in storage | No alert fired during the grace period |
| Low quota | The gap between usage and quota exceeded the protection threshold |
03 · IMPACT
When no one can prove who they are
Credential issuance and metadata queries failed for all Google accounts, and Google could not verify request authentication, returning 5xx errors on practically all authenticated traffic. The official summary reports 47 minutes of unavailability for services that required Google OAuth.
The identity service stops verifying credentials for all products.
Authenticated Workspace apps go down, because they depend on accounts for login and access control.
In Cloud Console, anyone not authenticated cannot sign in; anyone who was may have degraded features.
Configurations already applied in the data plane keep operating; only operations that modified the data plane network were affected.
| Service | What happened |
|---|---|
| Cloud Console | Those already authenticated could keep using it, with degraded features |
| Rede | Only changes to the data plane VPC network were affected; existing configurations kept operating |
| Gmail | Some users saw errors for up to an hour after recovery, due to error caching in the identity services |
04 · DETECTION AND RECOVERY
Fast alert, fix by turning off the quota
The table below gathers the detection and recovery milestones reported by Google, in Pacific time.
| Time | Evento |
|---|---|
| 03:43 | Automatic capacity alerts |
| 03:46 | Errors begin in the User ID Service |
| 03:48 | Engineers paged |
| 04:08 | Root cause and possible fix identified |
| 04:22 | Quota enforcement turned off in one data center |
| 04:27 | Same mitigation applied in all data centers |
| 04:33 | Error rates back to normal |
05 · LESSONS
What Google committed to change
The report lists several actions. Among them: reviewing the quota automation to prevent rapid rollout of global changes; improving monitoring and alerting to catch incorrect configurations earlier; evaluating greater resilience to write failures in the identity service's database; and limiting the impact on the data plane during service failures.
Frequently asked questions
What is Google Dec 2020: quota reported as zero takes down global login?
Real incident. Key points: authentication and authorization.
What does the Google Dec 2020: quota reported as zero takes down global login design cover?
It is organized in these sections: Identity as a dependency of everything, A quota that believed in zero, When no one can prove who they are, Fast alert, fix by turning off the quota, and What Google committed to change.
How long is Google Dec 2020: quota reported as zero takes down global login and what level is it?
It takes about 15 minutes to read and is rated Staff level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.