Slack Jan 2021: the network gateway (AWS Transit Gateway) that did not scale when everyone returned from vacation

01 · CONTEXT

Back from vacation

For many people, the Slack outage happened on the first business day of the year, January 4, 2021. The team was paged by an external monitoring service because error rates were rising.

Slack's traffic is lower during the holidays. On the first Monday after them, client caches are cold and each client downloads more data than normal when connecting, so the company goes from the quietest time of the year to one of its biggest days practically overnight.

The systems that serve users scale quickly for this kind of spike and had always gotten through this post-holiday return without problems in previous years.

02 · THE TIMELINE

From error rate to unavailability

While the team was in the first moments of the investigation, the dashboard and alerting service became unavailable. Without the usual dashboards, debugging was impaired, but there were still internal consoles, command-line tools and logs.

Slack has mini-spikes of traffic at the start of every hour and every half hour, and the web tier and backends are scaled for them. The 7:00 mini-spike, added to the network problem that already existed, saturated the web tier: packet loss raised the latency of calls to the backends and Slack became unavailable.

03 · THE AUTOSCALING

Scaling up also needs health

The web tier scales on two signals: CPU usage and Apache worker thread usage. With the bad network, threads spent more time waiting and CPU dropped, which triggered an automatic downscaling. Right after, a significant upscaling came, because thread usage rose as the network got worse.

1. With the degraded network, the web tier waits longer for responses from the backends and threads spend more time waiting.

2. Autoscaling sees CPU drop, because threads are waiting, and triggers a downscaling.

3. Right after, thread usage rises and the company tries to add 1,200 servers to the web tier between 7:01 and 7:15.

4. The provision-service talks to the AWS APIs and internal systems over the same degraded network and, with so many instances at once, hits two resource bottlenecks.

5. Many instances are created without being fully provisioned and without serving traffic, and the broken instances lead to hitting the autoscaling group limits.

6. Around 8:15 the provision-service comes back and healthy instances start entering service.

Without the dashboards, several engineers were logged into production instances investigating. Many SSH sessions were suddenly terminated because those instances were deprovisioned, which made the investigation even harder; downscaling was then disabled to ease the investigation and preserve serving capacity.

04 · HOW IT CAME BACK

Panic mode, retries and circuit breaking

The load balancers' health checks kept failing at a very high rate because of the network. Load balancers have a panic mode that spreads requests across all instances when many fail health checks, and this, together with retries and circuit breaking, got the system serving again.

1. Clients send requests to the load balancers, which run health checks on the web instances.

2. When many instances fail health checks, the load balancer enters panic mode and starts distributing requests across all of them.

3. This mode, together with retries and circuit breaking, got Slack serving again, slower and with more errors than normal.

Around 9:15 Slack was degraded, not down. It took an hour to provision enough instances and bring errors down to a low level, for two reasons: the network was still not healthy, so more instances than normal were needed, and provisioning itself took longer than usual.

05 · THE CAUSE

The Transit Gateway that did not scale in time

By the time Slack had recovered, AWS engineers had found the trigger: part of AWS's network infrastructure became saturated and was dropping packets.

Slack uses AWS Transit Gateways (TGWs) as hubs to connect its VPCs. TGWs are managed by AWS and built to scale transparently for the customer, but this time they did not scale fast enough. AWS engineers were alerted by their own monitoring and increased capacity manually.

1. The Transit Gateway connects different VPCs; the dashboards and the supporting databases were in separate VPCs.

2. Because they were in different VPCs, the dashboards depended on the Transit Gateway to talk to their databases.

3. When the Transit Gateway became saturated and started dropping packets, the dashboards and alerts failed.

06 · WHAT CHANGED

Lessons and announced fixes

When designing a system for a predictable spike, it is worth checking the network, the provisioning tools and the observability dashboards too, and not only the number of servers. Asking a provisioner for too much capacity at once can overload it; limiting the pace of requests is a form of load control.