AWS Kinesis 2020: OS thread limit after adding capacity

01 · CONTEXT

A fleet in front of cells

On November 25, 2020, a failure in the AWS Northern Virginia region (US-EAST-1) affected Kinesis, the stream data processing service, and several services that depend on it. The case is worth studying for what AWS's official summary shows: the trigger was a routine operation, and the cause lay in a limit that nobody was watching. It connects with availability and fault tolerance, queues and streams e observabilidade.

Kinesis has two tiers. At the bottom, many back-end cells process the streams: each one owns several shards and works as a unit of scale and fault isolation. In front sits a front-end fleet that authenticates, rate-limits, and routes each request to the right shard. This fleet is what keeps, on each server, a cache called the shard-map.

1. Customers and other AWS services write and read records through the front end.

2. The front end routes each request to the back-end cell that owns the shard. The cell is the unit of scale and fault isolation.

3. To route, the server consults the shard-map, built from membership, configuration, and message data from the other servers.

4. Each server opens one operating system thread for every other server in the fleet. This detail is where the incident originates.

02 · THE TRIGGER

New capacity, hidden limit

Between 2:44 and 3:47 a.m., Pacific time, AWS added a relatively small amount of capacity to the front end. According to the summary, this was the trigger, not the root cause. The servers already running take up to an hour to discover the new ones and open the corresponding threads.

The first alarms for errors writing and reading records fired at 5:15 a.m. The cause, confirmed later, was that the new capacity made all the servers in the fleet exceed the maximum number of threads allowed by an operating system setting. Without threads, the cache could not finish being built, and the server was left with a useless shard-map, unable to route to the back end.

1. The fleet receives new servers. It looks like a safe, small change.

2. Each existing server gradually discovers the new ones and opens more threads, one per pair.

3. On every server, the total thread count goes past the operating system limit.

4. Without enough threads, the cache doesn't finish being built. The shard-map becomes unusable.

5. The front end doesn't know which cell to send the request to. The cells stay up, but become unreachable.

Each server's total thread count grows in proportion to the fleet size. Adding servers, which normally gives headroom, here consumed the resource that nobody was measuring. This is a per-server scale limitation, not a total-capacity one.

03 · THE DIAGNOSIS

Many errors, one cause

Diagnosis was slow because there were errors in every aspect of the calls made by new and old servers, which made it hard to separate effect from cause. The new capacity was suspect, and the team began removing it as a precaution. By 7:51 a.m. a few hypotheses remained, and all the most likely ones required restarting the entire fleet, a long and careful process. The cause was confirmed at 9:39 a.m.

The restart could not be quick. The resources on a server used to build the shard-map compete with those used to serve requests. Bringing servers back up quickly would create contention, leave few resources for serving, and cause slow servers to be treated as unhealthy and removed from the fleet, which would delay recovery.

04 · THE RECOVERY

Come back slowly to come back for good

With the new capacity already removed, the thread count would no longer exceed the limit, so the team chose to restart without raising the operating system limit. The first servers began receiving traffic again at 10:07 a.m. Because the fleet has many thousands of servers, only a few hundred per hour could be added, and Kinesis only returned fully to normal at 10:23 p.m.

1. A restarted server must build the shard-map and serve requests with the same resources.

2. If many come back at the same time, serving becomes slow. Slow servers may be considered unhealthy and removed, which delays recovery.

3. In the original method, the server gets shard-map data from its neighbors, which are also recovering.

4. In parallel, the team configured the servers to get the data directly from the authoritative metadata store, speeding up bootstrap.

05 · THE CASCADE EFFECT

Whoever depended on Kinesis went down with it

Kinesis is used by several other AWS services, and the failure spread through dependencies that, on paper, were tolerable. Cognito, for example, uses Kinesis only to collect access patterns, on a best-effort basis, with a local buffer. The prolonged failure triggered a latent bug in that buffer, and Cognito's web servers began to block. External users could not authenticate or obtain temporary AWS credentials.

1. Kinesis becomes degraded and every service that sends data to it starts to suffer.

2. Cognito sent data on a best-effort basis, but the latent bug in the buffer made it block on the backed-up queues.

3. The tool that publishes the public status dashboard depended on Cognito and was unavailable at first.

4. CloudWatch, which uses Kinesis for metrics and logs, lost most of them and its alarms went to an insufficient-data state.

5. Lambda buffered metrics until it caused memory contention, and reactive AutoScaling, without metrics, delayed scaling decisions.

6. EventBridge delayed events, and ECS and EKS, which use them in internal flows, saw impact on clusters and tasks.

06 · WHAT CHANGED

Fewer threads, more cells, better warning

In the short term, AWS decided to use servers with more CPU and memory, reducing the total number of servers and the threads on each. It also added fine-grained alarms on thread consumption and began testing a raise of the operating system limit. For cold start, it would move the front-end cache to a dedicated fleet and take large services, such as CloudWatch, to a separate, partitioned front-end fleet.

In the medium term, AWS promised to speed up the cellularization of the front end, to match what already existed in the back end. According to the summary, cellularization isolates the effects of failure and keeps components, like the shard-map cache, in an already-tested range. The work was already under way, but it was large and not yet finished.

The dependent services changed too. Cognito was modified to withstand Kinesis errors without exhausting its buffers, and CloudWatch began storing three hours of metrics locally, accessible to AutoScaling. AWS also revised training so that operators use the status dashboard's backup tool, which proved more manual and less familiar.