Imbue: 70B on 4,092 H100s

01 · CONTEXT

A cluster where everything has to work

Imbue trained a 70B-parameter model from scratch on its own infrastructure, with a small team, and published a guide that goes from bringing up the cluster to automatic error recovery during training. The guide is the source for this case.

The cluster described had 4,088 H100 GPUs spread across 511 computers, with eight GPUs per computer. Each GPU connected to its own ConnectX-7 card, capable of sending and receiving at the same time at 400 Gbps to any other GPU on the InfiniBand fabric.

The InfiniBand fabric had a three-tier architecture. Training communication went through it, not through Ethernet, which was used to transfer datasets, checkpoints, and other data.

1. The cluster is made of hosts with GPUs; each GPU has its own network card to talk to any other GPU on the fabric.

2. Training traffic between GPUs goes over the InfiniBand fabric, organized in three tiers.

3. Datasets, checkpoints, and other data travel over an Ethernet network, not over InfiniBand.

4. The InfiniBand design is centralized, with a single brain for the network; the team dealt with a single entity for the fabric's 320 switches.

5. A secondary Ethernet network serves only for configuration and management. The BMC lets you interact with each machine as if you were physically present.

02 · CLUSTER BRING-UP

From bare machine to InfiniBand fabric

A first server was installed manually, and Ubuntu's MAAS software helped provision the rest.

Once the wiring problems were solved, UFM was able to reach all the fabric's switches. After the recabling, the team added checks to confirm that the physical connections matched the new design.

03 · BURN-IN AND RELIABLE SET

Stressing the fabric and growing a set of reliable machines

To diagnose InfiniBand problems more efficiently, the team created a workload that pushed as much data as possible through all the fabric's ports at the same time. UFM alerted on transmission above 97% of theoretical capacity on most ports, and some switches went down temporarily. The ports still standing at the end of the day were considered robust enough; the rest were disabled or sent for repair.

This burn-in has the spirit of chaos engineering: exercising the infrastructure under deliberate stress, instead of waiting for the failure to show up on its own.

As a rule of thumb for new hardware, the source cites about 3% of machines breaking per week. The nuance is that this is not a uniform chance of failure per machine: a few problematic machines break repeatedly in different ways until they are fixed. So, instead of hunting for failures on random machines, the team could focus on growing a set of reliable machines.

04 · HEALTH AND RECOVERY

Per-host health checks and automatic relaunch

Most training problems could be attributed to faulty machines or network components, and it was essential to automate disabling them and requesting repairs. The health checks were designed to have a single front door, which answers yes or no per host.

1. A hardware failure, such as an Xid or SXid error, can bring down the run without producing a useful Python stack trace.

2. The orchestrator runs the health checks on all available machines and ranks the health of each one according to the checks it passes.

3. Each host's result boils down to a yes-or-no answer.

4. The orchestrator relaunches the training on the healthiest machines, from the most recent checkpoint.

5. For the network, a script reads UFM's event log, disables the implicated links and ports, opens maintenance tickets, and re-enables the components when the repair is done.

6. Having more machines than necessary, 10% to 20% more according to the source, makes it easier to relaunch when one of them fails.

Reserving machines beyond what is needed is a form of capacity redundancy.

05 · DIAGNOSIS

Hangs without stack traces and MFU slowdowns

The most memorable hang showed up at the same time in all of the run's workers: when a host did not complete an NCCL operation, the others stayed blocked on it until NCCL_TIMEOUT. The nature of NCCL made it very hard to find the culprit host. The team changed the library's logging to show which operations were in flight at the time of the failure and, to find problematic hosts, often had to see which hosts did not produce certain log messages.

There were situations where a run on a set of machines failed without anyone knowing which one was the culprit. A group of 48 machines, for example, was split into six groups of eight and then into eight groups of six; the machine present in the failing runs in both rounds was the problematic one.

1. A group of machines fails and it is not clear which one is the culprit.

2. The group of 48 machines is split into six groups of eight, each running a smaller run.

3. Then the same set is split into eight groups of six, with new smaller runs.

4. The machine that takes part in failing runs in both rounds is the problematic one.

For slowdowns, the source uses MFU (model FLOPs utilization) as the measure, and generic slowdowns or an MFU below what had already been demonstrated could have several causes. The shape of the MFU curve helped diagnose the class of the problem.

06 · LESSONS

What the case teaches about operating thousands of GPUs

The team highlights having 10-20% more machines than needed so it can relaunch easily when something fails, and writing tests and automatic fixes for every type of failure, because every problem found comes back. It also adopted the rule of changing one thing at a time.

Being a small team, it tried to automate as much of the process as possible.