Stripe: scaling the API with rate limiters

01 · CONTEXT

Why limit a production API

The first few times traffic spikes suddenly, it is reasonable to just add infrastructure capacity. In a production API, however, you also have to ensure that a bad actor does not affect the disponibilidade for everyone else, on purpose or by accident.

According to Stripe, limiting traffic helps in scenarios like these: one user causes a spike and the others must keep being served; one user sends many low-priority requests that cannot get in the way of high-priority ones; something fails internally and low-priority requests must be dropped.

Stripe distinguishes two mechanisms. A rate limiter is appropriate when users can change the pace of their calls without changing the outcome. Load shedding, in turn, drops low-priority requests during incidents so critical ones get through, and decides based on the state of the whole system, not on who made the request.

02 · THE FOUR LIMITERS

Limiters and load shedders working together

Stripe operates four kinds of limiters in production. The request rate limiter is by far the most important and the recommended starting point.

The request rate limiter restricts each user to a maximum number of requests per second. Stripe allows briefly exceeding the cap for sudden spikes during real-time events, and keeps the same limiting behavior in test and production modes.

The concurrency limiter is about how many requests can be in flight at the same time, rather than how many per second. Some endpoints consume far more resources, and users frustrated by the wait try again, adding demand to an already overloaded resource. Before adopting it, Stripe frequently dealt with contention on the most expensive endpoints; according to the post, it solved the problem completely.

The fleet usage load shedder separates traffic into critical methods, such as creating charges, and non-critical ones, such as listing charges. A Redis cluster counts the requests of each type, and a fraction of the infrastructure is always reserved for the critical ones, so non-critical requests above their share are rejected.

The worker utilization load shedder is the last line of defense. If a machine is too busy, it starts dropping less critical requests, beginning with test traffic, and escalates to more traffic if that is not enough. Dropping and restoring traffic must be slow, otherwise the system oscillates between the two states. According to the post, it limits the impact of incidents that are already happening, while the first three are preventive.

1. The client's request reaches the limiter layer, which decides whether it goes ahead.

2. The layer checks the user's token bucket in Redis and takes one token for each request.

3. With an empty bucket, the request is rejected: this is the rate limiter acting on that user.

4. A Redis cluster also counts the requests of each type; non-critical ones above their allotted share are rejected so capacity is left for the critical ones.

5. On the workers, if the queue tightens, the last line of defense starts dropping less critical traffic, beginning with test mode.

03 · IMPLEMENTATION

Token bucket in Redis

Stripe uses the token bucket algorithm, with a central bucket host: each request takes a token, and new tokens drip into the bucket gradually. With an empty bucket, the request is rejected. Each user has their own bucket, and the implementation uses Redis, which can be self-operated or, on AWS, be a managed service such as ElastiCache.

When rejecting, you must choose between HTTP 429 and HTTP 503 depending on the situation, and the message returned should be actionable for the caller.

04 · SAFE ROLLOUT

Introducing limiters without breaking users

A new limiter can block people who should not be blocked. Stripe recommends kill switches to turn it off if it acts by mistake, with feature flags as a human escape hatch, plus alerts and metrics to know how often it triggers.

It also recommends a dark launch of each limiter, to observe the traffic it would block. The idea is to find thresholds that keep the API up without affecting users' existing request patterns, which may involve working with some of them to change their code.

1. On the normal path, the middleware queries Redis before deciding the request's destination.

2. Imagine the limiters' Redis goes down or there is a bug in the limiting code.

3. The middleware catches the exception and fails open: the request proceeds to the service and the API stays functional.

4. In case the limiters act by mistake, feature-flag kill switches turn them off, and alerts and metrics show how often they trigger.

05 · LESSONS

Where to start

According to the post, not all of the strategies are needed on day one. Start with the request rate limiter and introduce the other three over time, to prevent different classes of problems, as the need arises.