Incident · Staff
AWS us-east-1: internal network congestion
A spike in connections on the internal network affected services and the status dashboard itself.
Short answer
A spike in connections on the internal network affected services and the status dashboard itself. In December 2021, the AWS us-east-1 region suffered a long event that hit many services. The official post-mortem describes two networks: the main network, where customer services run, and an internal network, which supports operational tools such as real-time monitoring data.
01 · CONTEXT
Two networks, one fragile bridge
In December 2021, the AWS us-east-1 region suffered a long event that hit many services. The official post-mortem describes two networks: the main network, where customer services run, and an internal network, which supports operational tools such as real-time monitoring data. Between them are network devices that act as a bridge. It all started on the main side, with an automated capacity-scaling activity of one of the AWS services.
In normal operation, the services on the main network talk to the internal network through network devices dedicated to this bridge.
The internal network hosts monitoring and other services that operations depends on to see and control the platform.
02 · TRIGGER
From automatic scaling to a retry storm
The scaling activity generated a large spike in connections that overloaded the devices between the two networks. The overload increased latency and errors, which caused even more connection attempts and retries, in a feedback loop. The post-mortem also points to a latent problem that kept these clients from backing off properly during the event. It is the classic design of missing overload control: when the client insists more precisely because the server is slow, the extra load keeps the bottleneck alive.
An automated scaling activity triggers inside the main network.
The connection spike overloads the devices that link the main network to the internal network.
Latency and errors generate new attempts; without proper backoff, the retries add load and congestion persists.
03 · PLANES
Control plane and data plane
The impact was uneven. According to the post-mortem, EC2 instances that were already running were not affected, and customers accessing S3 and DynamoDB were not impacted by the event. Those who suffered were the operations that depend on the congested path, such as creating new resources and calling control APIs. This separation between what is already running and what needs coordination is what makes multi-region architecture and patterns of disponibilidade so important: the data plane usually holds up better than the control plane.
| Camada | What the post-mortem reports | Reading |
|---|---|---|
| Running EC2 instances | Not affected by the event. | Data plane preserved. |
| Access to S3 and DynamoDB | Customers accessing these services were not impacted. | Dependency outside the congested bridge. |
| EventBridge event delivery | Operators disabled event delivery during the response. | Deliberately turning off a function to relieve pressure. |
Workloads that were already running keep serving the customer, without depending on the congested path.
Direct access to object storage and to the NoSQL table was not impacted.
Control APIs depend on the bridge between the networks and become slow or error out while it is congested.
To relieve pressure, event delivery was disabled by operators until the situation improved.
04 · RESPONSE
Operating blind
The congestion immediately affected the availability of real-time monitoring data for internal teams. Without good visibility, diagnosing and acting became slower: the very tool for seeing the problem depended on the path that was saturated. AWS immediately disabled the scaling activities that originated the event. Even so, the post-mortem describes persistent congestion on the devices between the networks, which prolonged the impact. This is the central point of monitoring and observability: the observation channel must not share the bottleneck with what it observes.
05 · LESSONS
What to take to your project
Among the announced changes is a new support architecture that runs actively in several AWS regions, meaning the customer help path would no longer depend on a single region. From the standpoint of disaster recovery, it is the lesson of not putting the communication and support channel inside the same failure domain as the system it covers.
Before, support ran in a single region and went down along with it.
With an active multi-region architecture, customer traffic can go to the healthy region.
Frequently asked questions
What is AWS us-east-1: internal network congestion?
A spike in connections on the internal network affected services and the status dashboard itself.
What does the AWS us-east-1: internal network congestion design cover?
It is organized in these sections: Two networks, one fragile bridge, From automatic scaling to a retry storm, Control plane and data plane, Operating blind, and What to take to your project.
How long is AWS us-east-1: internal network congestion and what level is it?
It takes about 13 minutes to read and is rated Staff level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.