Training Data Centers: Synchronous, Dense, Checkpointable

During a 54-day stretch of Meta's Llama 3 405B pre-training run, up to 16,384 H100s were held in lock-step on a single job, the whole run moving at the speed of its slowest straggler. That is what a pre-training cluster is: one machine, a single tightly coupled supercomputer, not a fleet of independent servers that happen to share a building. That one fact reorganizes every downstream decision. The design objective is goodput, the fraction of wall-clock time the cluster spends doing useful gradient work, measured against the megawatts you can energize and the depreciation clock already running on the silicon. The facility must also meet your maintenance and fault requirements. Everything in this chapter follows from optimizing goodput per megawatt on a job that cannot tolerate a slow link, a hot GPU or an unsynchronized step.

This is the engineering treatment of the training-shaped side of the fork in Chapter 1.1. It takes the four defining properties of the workload (synchronous, dense, collective-dominated and checkpointable) and derives the building from them: the parallelism regime and its collectives; the scale-up domain and scale-out fabric that carry them; the power density and cooling the chosen rack requires; the reliability philosophy that checkpointing makes rational; the storage that feeds the GPUs and absorbs checkpoints; and the multi-datacenter and gigawatt-campus regime where a run outgrows one building. Accept the cluster only against a job-completion deadline that includes checkpoint writes, lost progress, restore, data input and collective stalls. A peak-rate test alone does not prove it.

Training a frontier model is a single optimization loop run across an enormous machine, and the model and its data are split four ways at once. Data parallelism replicates the model and shards the batch. Every replica computes gradients on its slice, and the replicas must agree before the next step, an all-reduce of the full gradient on every iteration. Tensor parallelism splits individual matrix multiplications across GPUs within a tightly coupled group, generating all-reduce and all-gather traffic on the critical path of every layer. Pipeline parallelism splits the layers into stages across nodes, passing activations forward and gradients back, and lives or dies on how small the pipeline bubble stays. Expert parallelism, for mixture-of-experts models, routes each token to a subset of experts on different GPUs, an all-to-all shuffle that is now one of the heaviest collectives in modern training. Together these are called 4D parallelism (5D with context/sequence parallelism), and they are why a pre-training cluster behaves as one organism rather than many.

One consequence governs the whole design: the job is synchronous, so it advances at the speed of the slowest participant. A single GPU running 10% slow (a thermal throttle, a flaky NVLink, a degraded optic) can slow the entire cluster’s affected critical-path phase by 10%, because every other GPU waits at the next collective barrier for the straggler. The whole-step penalty depends on that phase's share of the step and how much of it overlaps compute, so it is not automatically a 10% loss over the run. This is the straggler tax, and it is why training facilities are engineered for uniformity and tail control rather than average performance. It is also why one hardware failure does not degrade the job gracefully: it halts it, forcing a restart from the last checkpoint. Elastic membership or checkpointless recovery avoids that, but buys graceful degradation with replicated state and spare capacity.

Synchrony also shapes the electrical load. Aligned compute and collective phases produce correlated power swings across the hall. Production telemetry published by Microsoft, OpenAI and NVIDIA in August 2025 showed oscillation energy concentrated at 0.2–3 Hz (arXiv 2508.14318). Check the training load against the utility's ramp and oscillation limits; Chapter 4.5 derives the smoothing and storage choices.

AI clusters have two networks, and conflating them is the most expensive networking mistake in training. The scale-up fabric (NVLink and its NVSwitch fabric inside a node or rack) is the memory-coherent, ultra-high-bandwidth domain where GPUs talk as if they shared one address space. The scale-out fabric (InfiniBand or RoCEv2 Ethernet across the back end) connects those domains into the full cluster. The per-GPU bandwidth gap between them is roughly an order of magnitude. NVLink 5 provides 1.8 TB/s per GPU bidirectional, or ~900 GB/s each way (a 72-GPU NVL72 rack aggregates ~130 TB/s). For GB200, one 400G CX-7 scale-out NIC per GPU provides ~50 GB/s each way, giving a ~18× ratio. GB300 with one 800G CX-8 NIC per GPU, at ~100 GB/s each way, gives ~9×. Vera Rubin's current product specification gives 3.0 TB/s bidirectional NVLink and 1,600 Gb/s one-way scale-out per GPU, a ~7.5× directional ratio. These are endpoint bandwidth ratios; collective throughput depends on topology and traffic. Dividing the 1.8 TB/s bidirectional figure by a one-way NIC rate would give a misleading ~36×, so compare like with like, per direction. The first principle of training-fabric design follows: keep the heaviest collectives inside the scale-up domain, because every byte pushed onto scale-out costs an order of magnitude more in bandwidth and latency.

Scale-up domain size is therefore a first-class workload decision. An 8-GPU HGX node, a 72-GPU NVL72 rack and the coming 576-GPU Rubin Ultra NVL576 domain across eight MGX racks are not just bigger boxes. Each enlargement raises the ceiling on how much tensor and expert parallelism fits before spilling onto scale-out. A larger NVLink domain keeps a whole tensor-parallel group, or a wider set of MoE experts, inside the cheap fabric, which is why the industry is racing domain size upward. The downside is blast radius and packaging. An HPE GB200 NVL72 rack weighs approximately 3,245 lb fully loaded with PGW coolant (QuickSpecs V6, September 2026), and NVIDIA describes Vera Rubin NVL72 as roughly 4,000 lb (March 2026). The GB200 NVL72 copper spine carries 5,184 in-rack NVLink cables, and a single NVSwitch fault now degrades 72 GPUs instead of 8.

Published scale-out fabrics span 1:1, 2:1–3:1 and reported 7:1 examples. Treat those as observations, not training or inference defaults. Chapter 8.5 derives the per-tier ratio through three tests (locality, exposure and failure headroom, all read off the measured traffic matrix) and validates it on the target fabric. Synchronous collectives often justify high bisection, and local inference often permits upper-tier oversubscription, but distributed MoE inference, KV movement and prefill/decode disaggregation can demand more. SemiAnalysis's October 2024 model estimated 31.6% back-end savings from combining virtual modular switch placement with 2:1 oversubscription; the non-blocking virtual modular switch alone saved 24.9% against the same reference. That is a modeled result, not permission to choose either branch without workload evidence. → Chapter 8.5 (topology, sizing, oversubscription); Chapter 8.2 (scale-up fabric); Chapter 8.4 (InfiniBand vs RoCE).

The accelerator generation and the scale-up domain you chose set density in a training hall, and they put you on one side of a discontinuity. An HPE GB200 NVL72 has a 132 kW nominal rack TDP. NVIDIA's reference facility design basis for GB300 is up to 142 kW, and the Rubin Ultra / Kyber facility planning point is ~600 kW. Check each against its published facility limits before it enters your design basis. For the HPE GB200 NVL72, roughly 115 kW goes to liquid and 17 kW to residual air, and that product record, not a universal rack-kW threshold, is what makes direct-to-chip liquid necessary. Choosing this dense rack therefore chooses liquid cooling, and it does so before you order a single GPU, because the slab loading, the facility-side heat-rejection path and the pipe-rack space all have to exist first. → Chapter 5.1 (the density wall); Chapter 5.4 (DLC, the 2026 default).

The same heat split, together with the rack's airflow and inlet limits, also fixes the water side. The rack's maximum liquid inlet and maximum liquid return are separate acceptance limits, not an operating pair. Flow follows from the approved fluid, the liquid load and the design ΔT you choose, not from either maximum. Chapter 5.1 owns that heat balance and Chapter 5.7 the fluid limits, including the ~165–236 L/min worked band for this rack's 115 kW liquid share. A rack outside its supported thermal range can throttle, so correlate facility and IT telemetry against your operating point. → Chapter 5.7 (warm-water loops); Chapter 5.6 (CDUs and the secondary loop).

Reliability posture is where a training facility most needs two distinct models. Facility availability measures whether power, cooling and connectivity stay inside their contracted states. Training goodput measures how much accelerator time advances the job after node failures, detection, checkpoint overhead, restart and replay. Checkpoint-and-resume lowers the consequence of some interruptions, but it does not make facility continuity irrelevant or select a topology. Model both.

The failure rates are not theoretical. Meta's Llama 3 405B run logged 419 unplanned interruptions in a 54-day snapshot of a run on up to 16,384 H100s, roughly one every three hours. Table 5 attributes 148 of the 419 events to faulty GPUs and 72 to HBM3 errors. SemiAnalysis reported roughly seven days of MTBF for one 512-H100 cluster at a top-tier operator in October 2024. That observation is not a per-GPU rate or a scaling law. Larger jobs expose more failure domains, but the interruption distribution for your fleet, job, software stack and event definition has to be measured. Alibaba's Unicron study (December 2023) reported abnormal termination in 43.4% of the top 5% most resource-intensive training tasks. In its GPT-3 recovery example on 256 H800 GPUs, transient faults accounted for 73% of errors and were typically recoverable by restarting (arXiv 2401.00134). At these rates the cluster is always healing, so the design question is not 'how do we prevent failures' (you cannot) but 'how do we make each failure cheap.'

That reframing changes the economics of the redundancy decision without making it automatic. Compare each defined maintenance and fault state, transfer interruption, post-event loading, path and control independence, common modes and recovery time against your uptime or recovery target. Checkpointing, hot spares and fast detection can make N, N+1 or distributed power economic for a given design, while a contract or recovery limit can still require uninterrupted paths. The anti-pattern is selecting either 2N or a lean topology from “checkpointable training” alone. → Chapter 12.2; Chapter 12.5.

The straggler economics close the loop. Because the job runs at the slowest GPU's pace, a partially degraded node is often worse than a dead one. A dead node is evicted and replaced, but a silently slow node taxes every step until it is detected. Mature operators therefore invest heavily in tail telemetry: per-GPU thermal and clock monitoring, NVLink and optic error counters, and collective-timing instrumentation that flags the straggler before it has bled hours of goodput. The facility's job is to give that telemetry nothing to find, with uniform cooling, uniform power and no thermal hot spots, because every degree of thermal non-uniformity across the hall is a latent straggler. → Chapter 14.2 (DCIM and telemetry); Chapter 10.6 (GPU health observability).

The optimal-interval mathematics, the Young/Daly result that sets the checkpoint cadence by balancing checkpoint cost against expected lost work, is canonical in Chapter 9.4. Here we cover only its bearing on the synchronous training building, which is twofold and concrete.

First, checkpointing changes the outage consequence and can strengthen the economic case for N or N+1, but it does not select the facility topology. The state-based model must still prove which maintenance and fault cases may interrupt load, for how long, at what post-event capacity, and with what recovery and common-mode exposure. The cheaper and faster the checkpoint, the less work a failure costs, the more aggressive a cadence you can afford, and the lower the goodput penalty of any given MTBF. That makes checkpoint bandwidth a first-class facility requirement. The storage system must absorb a full-cluster checkpoint, terabytes of optimizer and model state, fast enough that the GPUs stall only briefly, because every GPU is idle during a synchronous checkpoint barrier. A slow checkpoint path converts into lost goodput on every interval. → see storage, below, and Chapter 9.3 (GPUDirect Storage).

Second, the cadence interacts with the failure rate to size everything else. At one interruption every three hours (Llama 3 scale), a checkpoint cadence and a lost-work budget together determine how many hot spares you must keep warm, and how fast the orchestration plane must detect, evict and re-place a failed node to keep measured useful progress inside the job-completion budget. Provision the checkpoint storage tier and the spare-node pool as deliberately as you provision GPUs. They are the levers that convert a high failure rate into high goodput. → Chapter 10.7 (autonomous recovery); Chapter 14.6 (spares strategy).

A training cluster's storage keeps expensive GPUs fed and absorbs checkpoints without stalling them. It has three distinct jobs with different performance shapes. Checkpoint write bandwidth is bursty and enormous: at a synchronous barrier the whole cluster writes its state at once, so the storage must sink terabytes in seconds to minimize the idle window. Dataset streaming (the data-loader path) is sustained, read-heavy and latency-sensitive in the tail. If the loader cannot keep every GPU's input queue full, the GPUs starve and MFU drops, the same straggler logic applied to data instead of compute. Object/capacity storage holds the raw corpus and cold checkpoints. A parallel file system (Lustre, GPFS/Storage Scale, WEKA, VAST and kin) sits in front, and the data-loader and checkpoint paths increasingly use GPUDirect Storage to move bytes directly into GPU memory, bypassing the CPU bounce buffer. → Chapter 9.1 (why storage determines GPU efficiency); Chapter 9.2 (parallel file systems); Chapter 9.5 (the data-loader path).

LOSF (Lots Of Small Files) is the storage pathology specific to AI training, and it ambushes teams that sized for bandwidth alone. Training corpora and tokenized shards are frequently millions of small objects, and small-file workloads are bound by metadata operations and IOPS, not sequential throughput. A file system tuned for the big sequential reads of checkpoint restore can choke on the random small-file reads of dataset streaming, leaving GPUs starved while the bandwidth meter reads low. Specify the storage tier against the LOSF and checkpoint-burst profiles explicitly, not against an average GB/s number, and size the metadata path (often the silent bottleneck) as deliberately as the data path. → Chapter 9.8 (sizing and data gravity); Chapter 9.9 (offline data-prep).

A single building has a ceiling: the power it can energize, its land, a coherent cooling plant. Frontier runs have already hit it. Power is the binding constraint of 2026, and one campus increasingly cannot host enough of it for the largest runs, so the supercomputer is being stretched across multiple buildings and campuses. Google trained Gemini Ultra across TPU SuperPods in multiple data centers, model-parallel within a superpod and data-parallel across superpods, over its intra- and inter-cluster network, with latency and bandwidth sufficient to preserve a synchronous training paradigm. Microsoft gives the reason plainly for its Fairwater sites: land and power availability make it necessary to spread the work across locations. Meta's Prometheus in New Albany, Ohio is a regional cluster: its September 2025 plan targeted 1 GW across five or more data-center buildings, weatherproof tents and adjacent colocation facilities. Microsoft's Fairwater AI WAN describes the different case of synchronized training across regions. → Chapter 8.8 (scale-across: multi-campus and cross-region fabric).

This forces a real fork. Synchronous cross-DC training keeps the single-job, lock-step model and extends the non-blocking fabric across campuses with dedicated dark fiber. It preserves model quality and simplicity, at the cost of enormous, low-latency, high-bandwidth inter-campus links and the speed-of-light latency floor between sites, which bounds how far apart they can sit before the collective stalls. Asynchronous / low-communication training (the DiLoCo family: Streaming DiLoCo, DiLoCoX and async variants) lets each site take many local steps before exchanging compressed pseudo-gradients. That slashes inter-site bandwidth by orders of magnitude (DiLoCo demonstrated comparable quality while communicating ~500x less) and tolerates wide-area links and stragglers, at the cost of algorithmic complexity, staleness management and a model-quality regime that is still maturing. The choice rewrites the inter-campus fiber plan, the power-resilience design and the orchestration plane. → Chapter 8.8; Chapter 10.8 (training frameworks).

The recurring training mis-scopes share a root: reasoning from the equipment or the building instead of from the synchronous, dense, checkpointable nature of the job. Four are worth naming.

• Unvalidated back-end blocking. Copying 1:1, 2:1 or 3:1 from a workload label instead of measuring collectives, placement, overlap, failure headroom and the step-time target. This either overbuilds stranded bisection or starves the job. Validate every tier on the target fabric. → Chapter 8.5.
• Topology chosen from checkpointability. Treating “checkpointable” as proof that N/N+1 is enough, or as proof that 2N is waste, without modeling maintenance states, defined faults, transfer interruption, post-event loading, independence, common modes, recovery SLO and contract. Checkpointing is one consequence-mitigation input. → Chapter 12.2.
• Designing to today's density. Pouring a slab and water plant for the current generation, then being unable to absorb the next density step without structural rework, new utility service or tearing out plant under a running hall. Price the irreversible reserve for each named ramp scenario and record when it is released. → Chapter 5.10.
• Bandwidth-only storage spec. Sizing the storage tier against an average GB/s number while ignoring the LOSF metadata profile and the stop-the-world checkpoint burst, which starves GPUs on the data-loader path or stalls them on every checkpoint. → Chapter 9.1.