OpenAI: Kubernetes with 7,500 nodes

01 · CONTEXT AND WORKLOAD

A huge cluster for an unusual workload

In a public post, OpenAI described Kubernetes clusters with 7,500 nodes. It says itself that scaling a single cluster to this size is rarely done and requires special care.

The workload explains the choices. In many jobs, a single pod occupies the entire node; that is why there are many nodes but little pressure on the scheduler. That pressure, however, comes in bursts: a new job can create hundreds of pods at once and then churn goes back to low.

If a pod participating in a job dies, the whole job stops and has to be restarted, and it resumes from the last checkpoint. The pods talk directly by IP, with MPI over SSH, and not through service endpoints.

02 · API SERVER AND ETCD

The control plane as a critical component

OpenAI treats the API servers and etcd as critical components for a healthy cluster. The API servers run outside the cluster, and both they and etcd sit on dedicated nodes.

1. In the diagram, the researchers talk to the API servers, which are the central point of the control plane.

2. Kubernetes events live in an etcd cluster separate from the main one; after that separation, the team says it has had no notable problems with etcd.

3. In the largest clusters there are 5 API servers and 5 etcd nodes, to spread the load and limit the impact if one of them goes down.

4. The rate of 429 and 5xx responses from the API servers is a high-level signal of problems, and the team alerts on it.

5. Each API server's memory tends to grow linearly with the number of nodes; in the 7,500-node cluster, it reaches up to 70GB of heap per API server.

6. The API servers are stateless and easy to run in a self-healing group; for etcd there is no self-healing automation, because incidents were rare.

Separating state (etcd) from the stateless layer (API server) lets you treat each part differently: disposable replicas in front and a store that deserves greater care behind. It is the same reasoning as making disposable scaling horizontally what is stateless and protecting more rigorously what holds state replicated by consenso.

03 · LOAD THAT GROWS WITH THE CLUSTER

WATCHes, DaemonSets, and the API server

One of the biggest sources of load on the API server was the WATCH on Endpoints. Every node in the cluster is a member of services such as kubelet and node-exporter, and when a node joins or leaves, the WATCH fires.

1. Every node is a member of these services; when a node joins or leaves, the WATCH fires on the API server.

2. The WATCH on Endpoints was one of the big sources of API server load.

3. EndpointSlices reduced this load by 1000x.

4. The team keeps DaemonSets from interacting with the API server.

5. When every node needs to track changes, an intermediate caching service looks like a good pattern to avoid bottlenecks across the whole cluster.

04 · GANG SCHEDULING

All or nothing in scheduling

In experiments with StatefulSets, for the optimizers all members must be scheduled before any training, and MPI is sensitive to changes in group composition.

If two experiments ask for all the capacity, instead of scheduling one of them in full, Kubernetes may schedule only half the pods of each, and neither makes progress.

1. Two experiments each ask the scheduler for the cluster's entire capacity.

2. Kubernetes may schedule only half the pods of each experiment.

3. The result is a deadlock: neither experiment can make progress.

4. The solution adopted was the Coscheduling plugin in the scheduler.

The team tried custom scheduler solutions. Kubernetes 1.18 brought a plugin architecture to the core scheduler, which made it easier to add this kind of feature natively.

05 · ISOLATION AND LESSONS

Isolated clusters and what repeats

On the network, researchers connect to a hub and from it reach each cluster, the spokes. The clusters do not talk to each other, which preserves fault isolation.