Meta: 24k-GPU clusters for Llama 3

01 · CONTEXT

Two clusters to train Llama 3

Meta announced two 24k-GPU clusters, described as two versions of a data-center-scale cluster with 24,576 GPUs each. The post states that this cluster design is used to train Llama 3.

The starting point was the RSC, announced in 2022 with 16,000 NVIDIA A100 GPUs. According to the post, the new clusters build on the RSC's successes and lessons, with an emphasis on end-to-end systems and on researcher and developer experience and productivity.

1. The diagram starts with the researcher, whose training job goes through the scheduler before reaching the GPUs.

2. The internal job scheduler was changed to take the network topology into account when allocating jobs.

3. The GPU nodes use the Grand Teton platform, designed by Meta and contributed to the OCP.

4. Data and checkpoints go through a custom FUSE API on top of a Flash-optimized version of Tectonic.

5. Hammerspace's parallel file system makes code changes immediately accessible to all nodes, which enables interactive debugging.

6. Meta is building distributed debugging tools, such as desync debug and a collectives flight recorder.

02 · NETWORK

RoCE versus InfiniBand, side by side

Meta built one cluster with a RoCE (RDMA over Converged Ethernet) network, based on Arista 7800 switches with OCP Wedge400 and Minipack2. The other cluster uses an NVIDIA Quantum2 InfiniBand fabric. Both solutions interconnect 400 Gbps endpoints.

The stated reason for having both is to be able to assess the suitability and scalability of each type of interconnect for large-scale training. Through co-design of network, software, and model architectures, Meta says it successfully used both on large GenAI workloads, including the ongoing Llama 3 training on the RoCE cluster, without network bottlenecks.

03 · STORAGE AND CHECKPOINTING

Saving the state of thousands of GPUs

The post highlights storage as an important piece of training and as one of the least discussed aspects. Data and checkpoints go through a custom FUSE API, backed by a Flash-optimized version of Tectonic. The solution lets thousands of GPUs save and load checkpoints in a synchronized way, a challenge for any storage, and also provides exabyte-scale storage for the data workload.

1. Thousands of GPUs need to save and load checkpoints in a synchronized way, which is hard for any storage.

2. Access goes through the custom FUSE API and reaches a Flash-optimized version of Tectonic.

3. The same layer delivers flexible, high-throughput, exabyte-scale storage for the data workload.

4. According to the post, new storage solutions gave the opportunity to optimize checkpointing across thousands of ranks to run in hundreds of milliseconds.

5. Through the parallel file system, a code change becomes immediately accessible to all nodes, and the engineer debugs the job interactively.

In parallel, Meta partnered with Hammerspace for a parallel file system: code changes become immediately accessible to all nodes and the engineer debugs jobs with thousands of GPUs interactively. Together, Tectonic and Hammerspace enable fast iteration without giving up scale.

The storage servers use the YV3 Sierra Point platform with high-capacity E1.S SSDs, and the layer tolerates the infrastructure's daily maintenance operations.

04 · PERFORMANCE AT SCALE

From the small cluster to the large one

A principle stated by Meta is that the best way to test a design's scalability is to actually build the system, optimize it, and test it: simulators help, but only up to a point. The initial performance of the large clusters was poor and inconsistent compared to that of the optimized small clusters.

To close the gap, Meta changed the internal job scheduler to take the network topology into account, which brought latency gains and reduced traffic to the upper layers of the network. It also optimized the routing strategy together with changes to NCCL, aiming for ideal network utilization.

05 · OPERATIONS AND NEXT STEPS

Debugging, software, and the roadmap

The post treats debugging as one of the major challenges of large-scale training: identifying a problematic GPU that stalls an entire job becomes very hard. Meta says it is building tools like desync debug and a distributed collectives flight recorder, and that it reduced process group initialization time from, at times, hours to minutes.

Hardware also pushes software: H100 GPUs open the possibility of using new data types like FP8 in training. The stated goal is to end 2024 with 350,000 H100 GPUs, in a portfolio with power equivalent to nearly 600,000 H100s.