Google Cloud Jun 2019: automation deschedules the network control plane

01 · CONTEXT

A Sunday afternoon, congestion across several regions

On June 2, 2019, Google Cloud projects running services in multiple regions of the US suffered high packet loss from network congestion, for a period between 3 h 19 min and 4 h 25 min. The duration and degree of loss varied considerably from region to region.

The report lists the regions that saw higher latency, intermittent errors, and loss of connectivity: us-central1, us-east1, us-east4, us-west2, northamerica-northeast1, and southamerica-east1. Instances in us-west1 and in all regions of Europe and Asia did not experience regional congestion.

On the user side, Google reported that YouTube measured a 2.5% drop in views for one hour and that Google Cloud Storage measured a 30% reduction in traffic. Search, meanwhile, recorded only a brief increase in latency as it began serving from unaffected regions.

02 · HOW THE NETWORK WAS ORGANIZED

Logical clusters, control plane, and fail static

Within each physical location, Google's machines are separated into several logical clusters, and each has its own cluster management software. The design provides resilience to the failure of any individual manager.

The network control plane also runs under instances of the same cluster management software. In each location there are several instances, so the failure of one of them has no impact on network capacity.

This software plays a significant role in datacenter maintenance automation, such as changes to power infrastructure or network expansion. At global scale, maintenance is common, although rare at any single location. Each job is labeled with what it should do when maintenance happens: generally, be moved to a machine outside the maintenance or stop and be rescheduled later.

1. The cluster manager takes part in the automation of datacenter maintenance events.

2. The network control plane jobs run under instances of that same software, in each location.

3. Each job carries a label that says how to react to maintenance: be moved to another machine or stop and be rescheduled later.

4. The network was designed to keep working for a period even without the control plane present, an extra line of defense.

Google describes this design as defense in depth: even with a control plane designed to be highly resilient, the network enters fail static mode as an additional layer.

03 · THE CHAIN OF FAILURES

Two benign configurations and a bug

The report states that two normally benign configurations and a specific software bug combined to start the outage.

1. The maintenance event starts at a single physical location, and the automation builds the list of jobs to deschedule at that location.

2. The logical clusters on the list also contained network control jobs from other physical locations.

3. The automation deschedules each logical cluster on the list, including the network control jobs and supporting infrastructure in several locations.

4. Without the control plane, the network keeps operating normally for a few minutes.

5. After that period, BGP routing between the affected locations is withdrawn, network capacity drops sharply, and some regions become unreachable.

End-user impact began to appear between 11:47 and 11:49 (Pacific time), and engineers were alerted two minutes after the failure began.

04 · CONGESTION AND RECOVERY

When the tools share the network they need to fix

Debugging was severely hampered because the tools began competing for use of an already congested network. Google had robust contingency plans, but using them, which included engineers going to secure facilities and lowering the priority of less critical traffic classes, added time to debugging.

1. With reduced capacity, traffic tried to fit into what was left and the network became congested.

2. The network prioritized small, latency-sensitive traffic and dropped larger, less sensitive traffic.

3. The engineers' tools competed for the same congested network, which hampered debugging.

4. Teams from several services steered traffic away from the affected regions to keep serving from elsewhere.

The scope and scale of the outage, together with collateral damage to the tools, made it hard at first to identify the impact precisely and to communicate with customers.

At 13:01 (Pacific time) the root cause was identified and engineers stopped the automation responsible for the maintenance event. Recovery still had a second delay: with all the control plane instances descheduled in several locations, the configuration had been lost and had to be rebuilt and redistributed.

1. Engineers stop the automation responsible for the maintenance event.

2. With all the control plane instances descheduled, the configuration had been lost and needed to be rebuilt and redistributed.

3. As the control plane is rescheduled in each location and the configuration is recreated and distributed, network capacity returns.

05 · WHAT GOOGLE COMMITTED TO CHANGING

Limiting the reach of automation and speeding up recovery

The report describes the automation that deschedules jobs in the face of maintenance as immediately halted, and only reactivated after safeguards exist against descheduling jobs in several physical locations at the same time. The cluster manager will also be hardened to reject such requests, whatever their source.

For recovery, the control plane at each location would start persisting its configuration, so it need not be rebuilt and redistributed if all jobs are descheduled. The network would also remain in fail static for longer, widening the recovery window with no user impact.