Incident · Senior
Heroku/Travis 2022: stolen OAuth tokens
Real incident. Key points: authentication and authorization.
Short answer
Real incident. Key points: authentication and authorization. In April 2022, GitHub identified that user OAuth tokens, issued to two third-party integrators (Heroku and Travis CI), had been stolen and used to download data from dozens of organizations, among them npm.
01 · CONTEXT
Third-party tokens stored in a database
In April 2022, GitHub identified that user OAuth tokens, issued to two third-party integrators (Heroku and Travis CI), had been stolen and used to download data from dozens of organizations, among them npm. The case is a study of authentication and authorization in integrations: the sources are Heroku's incident report and GitHub's security alert.
According to Heroku, an attacker gained access to one of the company's databases and downloaded the GitHub integration OAuth tokens stored for its customers. Each of those tokens was a customer's integration credential for GitHub.
In the design, the user connects their own GitHub account to the integrator's OAuth app.
The integrator stores its customers' GitHub integration OAuth tokens in a database.
With a token, the app calls the GitHub API on the user's behalf.
The API gives access to the private repositories the token covers.
GitHub states that it does not believe the attacker obtained the tokens through a compromise of GitHub or its systems, because the tokens in question are not stored by GitHub in their original, usable format.
02 · THE ATTACK
From the archived repository to the customers' repositories
Heroku's investigation found that the initial access came from a compromised token for a machine account. The attacker obtained that access from an archived private repository containing Heroku source code, and the company assessed that the attacker reached the repository through a third-party integration. Heroku says it was unable to definitively confirm which integration was the source of the attack.
According to Heroku's assessment, the attacker reached an archived private repository with the company's source code through a third-party integration.
From that repository, the attacker gained access to a machine account; a compromised token for it was the initial access to the environment.
The attacker gained access to a Heroku database and downloaded the customers' stored OAuth tokens.
With the stolen tokens, the attacker authenticated to the GitHub API.
For most users, they listed the organizations, chose targets and listed the private repositories of the accounts of interest.
Then they cloned some of those private repositories.
03 · THE USE OF THE TOKENS
What the stolen tokens allowed
GitHub describes the attacker's pattern with the stolen tokens: they authenticated to the API and, for most affected users, listed the organizations and chose targets based on that list. They then listed the private repositories of the accounts of interest and cloned some of them.
In npm's case, the initial detection was unauthorized access to production infrastructure, made with a compromised AWS API key. GitHub believes this key was obtained when the attacker downloaded private npm repositories using a stolen OAuth token, and assesses that the attacker did not modify packages or access user account data or credentials.
Heroku also confirmed that, on the same day the attacker exfiltrated the OAuth tokens, the attacker downloaded data from another database, which stores pipeline-level configuration variables for Review Apps and Heroku CI.
04 · THE RESPONSE
Revoking everything has a cost
| Quem | Action described in the source |
|---|---|
| GitHub, with the integrators | Asked them to investigate, revoke all user OAuth tokens of the affected apps and notify their users |
| GitHub, internal use | Revoked the tokens associated with the compromised apps' internal use of GitHub and npm |
| Heroku | Revoked all existing tokens of the Dashboard's GitHub integration and prevented the creation of new ones |
| GitHub customers | Periodically review authorized OAuth apps and remove what is no longer needed |
Heroku itself acknowledges that cutting the GitHub integration and rotating credentials affected its customers, and records that it considered those actions a critical step to protect the data.
05 · FOR YOUR DESIGN
Less power per token, less damage
After the incident, Heroku said it believes that mTLS based on RFC-8705 and private key protection for OAuth, together with parity between the OAuth integration and the GitHub App model, provide more modular access privileges to connected repositories, and that it intends to explore those paths with GitHub.
An OAuth token is valid for whoever presents it. That is why some design questions weigh more than the hope of never leaking: is the token's scope the minimum necessary? Does it expire or can it be revoked quickly and in bulk? Who can read the database where it is stored?
Among Heroku's advice to customers is to review the integrations connected to their repositories and make sure they are trustworthy.
Frequently asked questions
What is Heroku/Travis 2022: stolen OAuth tokens?
Real incident. Key points: authentication and authorization.
What does the Heroku/Travis 2022: stolen OAuth tokens design cover?
It is organized in these sections: Third-party tokens stored in a database, From the archived repository to the customers' repositories, What the stolen tokens allowed, Revoking everything has a cost, and Less power per token, less damage.
How long is Heroku/Travis 2022: stolen OAuth tokens and what level is it?
It takes about 14 minutes to read and is rated Senior level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.