Rate limiter

01 · CONTEXT

Why limit the request rate

A rate limiter decides, for each request, whether it passes or is rejected. It protects the backend against spikes and abuse, and makes clear to the client how much it can consume. In an interview, the starting point is gathering requirements: limit per user, per IP, or per API key? What is the rule (how many requests per interval)? Does the limiter need to work with many servers? What happens when it fails itself?

The first decision is where it runs. On the client it does not work: requests can be forged by anyone with bad intentions, so the client is an untrustworthy place to enforce the limit. On the server side, the limiter can be a middleware or, most commonly, live inside the API gateway, which already sits in front of all the services.

1. The client is what originates the requests, but it cannot be what decides the limit: it is controlled by whoever wants to abuse the system.

2. The gateway receives all the traffic before the services; it is the natural place to apply the limit rule.

3. The counters for each key need fast storage: the gateway reads and updates an in-memory cache on every request.

4. Within the limit, the request proceeds to the service; above it, the gateway responds with the too-many-requests code and the service is never even touched.

02 · ALGORITHMS

Token bucket and leaking bucket

In the token bucket, each key has a bucket with a predefined capacity. Two parameters describe it: the bucket size, which is the maximum number of tokens it can hold, and the refill rate, the number of tokens put into it every second. Each request consumes a token; with no token, it is rejected. The bucket size defines how much burst is tolerated; the refill rate defines the sustained pace.

The leaking bucket is similar, but requests are processed at a fixed rate. It is usually implemented with a FIFO queue: what arrives enters the queue and leaves at a constant pace. This smooths the output, at the cost of not letting a burst through quickly.

03 · WINDOWS

Fixed window, sliding log, and sliding counter

The fixed window divides time into fixed-size intervals and keeps one counter per interval. It is simple and cheap, but has a known flaw: a burst at the edge between two windows can let through more requests than the quota allows, because the client exhausts one window at the end and the next one right at the start.

The sliding log stores the timestamp of each request and discards those older than the start of the current window; it solves the edge problem, but consumes a lot of memory, because even rejected requests may have their timestamps stored. The sliding counter is a hybrid of fixed window and log: it smooths spikes by using the rate of the previous window as a reference, with much less memory.

04 · DISTRIBUTION

Several gateways, one shared counter

With several gateway instances behind a load balancer, local counters are not enough: each instance would see only part of the key's traffic. The counter goes to a shared cache. Redis offers two commands that cover the case: INCR increments the stored counter and EXPIRE sets a time to live, after which the counter is deleted on its own.

The problem appears under concurrency: if two instances read the counter, increment it, and write it back without looking at each other, a count is lost and the limit is exceeded. Locks are the most obvious solution; the alternative is to run the whole decision in a Lua script, which is atomic, making the decision free of this race condition.

1. The load balancer spreads the requests for a single key across the gateway instances; none of them sees the full traffic.

2. Both instances now query the same cache; the counter stops being local and becomes shared state.

3. If both instances read the same value and each writes its own sum, an increment is lost and requests slip through above the limit.

4. With the read, the computation, and the write inside a single atomic script in the cache, the decision no longer has a window for a race.

05 · FAILURES AND RESPONSE

When the limiter goes down and what the client receives

The counter storage can also fail, and you need to decide in advance what to do. Fail-open lets traffic through: the cost is temporarily losing protection. Fail-closed rejects: the justification is that brief periods of rejected requests are preferable to a cascading collapse. There is no single answer; it depends on what the limit protects. Think about this together with availability patterns.

1. The counter cache becomes unavailable: the gateways can no longer read or update the limit state.

2. With fail-open, the gateways let everything through and the service is exposed to spikes without protection; with fail-closed, they reject and preserve the service at the cost of rejecting legitimate traffic.

When a request is rejected, the API returns HTTP code 429 (too many requests). Headers report the limit, the remaining quota, and how many seconds to wait before trying again, and a well-behaved client respects that time. To grow beyond one Redis node, one option is to hash the user identifier to choose the shard that stores that user's limit data.