Concepts
Token bucket and rate limiters
Intuition
Imagine a shared service counter where each person gets their own little basket of tokens: to be served, they hand over a token, and the attendant gradually refills the tokens in the basket. Someone using the counter at a normal pace never runs out of tokens, and someone who tries to rush through many times in a row empties only their own basket and waits, without getting in anyone else's way. When the whole counter is packed, another rule kicks in: service sets aside the less important requests so that the important ones keep being served.
How it works
In the token bucket described in the paper, a central host of buckets has a token removed from it on each request, while new tokens are gradually replenished; if the bucket is empty, the request is rejected. Since each user has their own bucket, a problematic agent (a buggy script, an accidental spike) exhausts only its own bucket and not the availability of the others. The gradual refill sets the sustained rate, and the tokens accumulated in the bucket allow absorbing a brief burst (the paper does not detail the bucket's capacity); the paper reports that, after analyzing traffic, Stripe began allowing the limit to be briefly exceeded during sudden spikes in real-time events, such as a flash sale. The concurrent requests limiter is a different thing: instead of counting calls per second, it restricts how many are in flight at the same time, which protects expensive endpoints, since users frustrated with the wait retry and add demand to a resource that is already overloaded. Load shedders decide based on the state of the whole system, not on the user who called: the fleet usage one reserves a share of capacity for critical methods and rejects non-critical traffic above its allocation, and the worker utilization one, the last line of defense, drops lower-priority traffic when workers pile up requests, starting with test-mode traffic. Dropping and readmitting load should be slow, to avoid flapping. In the implementation, limiters should fail open, that is, a bug or an outage of the Redis that holds the buckets must not affect requests, and each new limiter should go through a dark launch that only observes the traffic it would block; kill switches allow turning off a limiter that fires by mistake, alerts and metrics track how often it fires, and rejection uses HTTP 429 (Too Many Requests) or 503 (Service Unavailable), whichever is more accurate.
In practice
The paper recommends starting with the per-user request rate limiter, which it calls by far the most important, and introducing the others gradually, since not all of them are needed on day one. Use a rate limiter when the client can space out calls without changing the result; if it cannot, as with real-time events, what is usually missing is infrastructure capacity. Concurrency limiters and load shedders come in when there are expensive endpoints or a risk of incidents, and it is worth having kill switches, trigger metrics and clear error messages, choosing between HTTP 429 and 503 according to which is more accurate.
Use it in a design
Practice the cases that rely on this concept in the System Design school.