The AI-Cluster Reliability Rethink: Goodput vs Facility Availability

For sixty years the data-center industry optimized one number: availability — the fraction of time the facility is energized and cooled, measured over a declared window; Tier certification separately assesses topology outcomes. A Tier III site promises concurrent maintainability; a Tier IV site adds fault tolerance — topology guarantees, not the percentage-uptime folklore still quoted from pre-2009 Uptime documents (Chapter 12.1). That metric was correct for the workload it was built around: enterprise applications and web services where the unit of value is a transaction, an outage is a binary up/down event, and a single rack going dark is a contained, recoverable nuisance. Redundancy — N+1, 2N, block- and distributed-redundant power, dual cooling paths — exists to push that one number toward unity.

An AI factory breaks the assumption underneath the metric. A frontier training job is one tightly-coupled supercomputer running synchronously across tens of thousands of accelerators; a single failed GPU forces a restart-all job back to its last surviving checkpoint when it takes out a required rank; replica-group recovery needs its own state and quorum rules. The facility can be at 100.000% availability — every breaker closed, every CDU pumping — and the cluster can still be throwing away a fifth of the money you spent on it, because the GPUs are idle waiting on a straggler, replaying lost steps, or stalled mid-checkpoint. Goodput — the share of bought GPU-hours that becomes useful work — is the number that governs return, and therefore the one the redundancy budget should be optimizing.

Define the terms precisely, because the whole rethink lives in the gap between them. Facility availability is a property of the physical plant: the fraction of the observation window in which the declared power-and-cooling conditions hold at the specified load interface; the facility classification is a separate statement about assessed topology and operating outcomes. ML Productivity Goodput is a property of the workload. Google's formulation decomposes it as ML Productivity Goodput = Scheduling Goodput × Runtime Goodput × Program Goodput — resource availability × retained forward-progress time × effective FLOP utilization. Keep three labels apart, because they carry different denominators: Provider Goodput (scheduling × runtime) is the part a contract can commit; ML Productivity Goodput multiplies that by MFU; and Meta's ETTR, or effective training time, is a productive-time observation that excludes MFU entirely. With illustrative Scheduling Goodput of 100%, Runtime Goodput of 90% and Program Goodput (MFU) of 40%, Provider Goodput is 90% and ML Productivity Goodput is 36% — one fleet, two denominators and numbers more than a factor of two apart. Low Program Goodput cuts FLOP productivity while retained-progress time stays unchanged; the contract must say which metric and code obligations it covers. In the scheduling and runtime terms, badput includes accelerator init, JIT compilation, data-loading stalls, checkpoint save and restore, wasted progress replayed after a failure, and infrastructure recovery during restarts (Google Cloud, 2024–2025).

The two numbers diverge because most badput is invisible to the facility. When the building loses power, both availability and goodput drop — they agree. But the dominant losses in a real cluster are not facility outages. Meta's published Llama 3 405B snapshot recorded 419 unplanned interruptions over 54 days on 16,384 H100s — roughly one every three hours — of which the paper attributes approximately 78% to hardware and reports 58.7% as GPU issues, while its table counts and percentages do not reconcile (Meta, 2024). Those are cluster-side interruptions, not evidence of a building outage; the paper does not report facility availability for the snapshot. For the same GPUs, window and eligible-time denominator, facility availability is a ceiling on retained-work time, never a floor: you can keep every required facility service present and still throw away progress below it. That ceiling does not compare one site with a surviving regional fleet or an exclusion-adjusted contract. State the boundaries, then price the lost work; polishing the ceiling while the workload bleeds below it misallocates the redundancy budget.

The deepest consequence of the goodput reframe is that redundancy migrates out of the facility and into the silicon and the software. In the availability model every resilience dollar bought a redundant power or cooling path. In the goodput model mechanisms above the facility boundary compete for recovered GPU-hours per dollar: multi-tier checkpointing that cuts a training restore from tens of minutes to under a minute; a hot-spare pool with fast health-check and drain, so a failed node is swapped in minutes rather than a fabric re-cable; per-GPU capacitance, rack BBUs and facility BESS that ride through the millisecond-to-second transients the cluster's own load swings cause; and, for serving, replica and region capacity that masks a site event the way a second power path never could. The facility chain still owns the states none of those can bridge — a distribution-path fault that drops a whole hall, a coolant-flow loss that trips every rack on the loop — and it must be sized for them. For a checkpointable fleet whose measured losses are dominated by replay and capacity waits, the checkpoint tier and spare pool move ahead of a second utility path after required facility states close. Reprice the loss still left after each purchase, because two upgrades can recover the same GPU-hours.

The table is a spend-allocation guide, and it reads differently for the four fleets an operator actually runs. For frontier synchronous training the ranking is checkpoint tier, spare pool, cooling continuity, transient ride-through, then facility path redundancy — the job already restarts every few hours, so a rare hall event is a marginal addition to a loss the first two rows have bounded; the order holds while measured losses are dominated by replay and capacity waits, and because a required-rank loss interrupts the run either way, compare interruption frequency, durable progress and complete recovery before buying. For smaller checkpointable training the order holds and the facility rows fall further. For frontier rack-scale inference on a single site, flip it: there is no checkpoint to fall back on, the SLA is measured in tail latency, and a hall outage is unmasked unless a second region absorbs it — so cooling continuity and the second power path move to the top. For replicated enterprise inference across regions the fleet masks the site, provided a surviving replica also has weights, keys, session state and sufficient capacity, and the facility rows are bought to the contract's floor and no further. Two things hold across all four: loss of coolant flow is now the cheapest hall-wide outage to prevent, and the contract, not the workload label, sets the floor the facility must certify. Chapter 12.3 selects regional recovery capacity.

torchft illustrates recovery for supported replicated training arrangements. Qualify the actual algorithm and surviving group; its support does not extend automatically to every training stack.

The most under-appreciated consequence of the density ramp is that coolant flow became a dominant single point of cluster-wide failure, and the loop now operates on two different transient clocks. An air-cooled hall carried enormous thermal inertia: chilled-water volume, the air mass of the room, raised-floor plenum. A CRAH failure can leave time to fail over or intervene when coupled thermal mass holds inlet temperature inside its limit; measure that window for the actual failed component. With flow maintained after rejection loss, coolant and metal buffer the heat input. When flow stops, the distant stored water can no longer carry heat away from the chip: the rack's throttle and trip states arrive on the window the OEM's transient data and a controlled commissioning pump-drop test fix for the specific product — an air-hall runbook cannot supply that window.

This relocates the availability problem. A facility can hold a fault-tolerant power topology and still take the entire cluster down through a coolant-distribution-unit fault, a pump trip, or a control-loop oscillation, when the failed loop exceeds its qualified interruption limit; the assessment record must identify whether the in-rack path is included. The design-basis response is to make coolant continuity a first-class redundancy line: redundant pumps and heat exchangers in the CDU, UPS-backed pump power, isolation that lets one CDU fail without starving the loop, and a commissioning test that drops a pump at full load and proves the ride-through — the posture behind the fleet-wide CDU availability Google has reported since 2020 (Chapter 5.11). Concurrent maintainability — the Tier-III property the industry already values — has to be re-earned in the liquid path: you must be able to pull a pump or service a heat exchanger without dropping the rack. Skimp on cooling-loop redundancy and you have built a cluster whose availability is capped by its weakest pump, no matter how many nines the power chain carries. → Chapter 12.1 sets the topology vocabulary; the DLC continuity engineering is in Chapter 5.4.

The reliability rethink runs in both directions. The cluster's own reliability depends on the facility — but the facility has become a reliability problem for the grid, and that coupling now feeds back into the cluster's design-basis. AI training loads are phase-coherent and synchronized: tens of thousands of GPUs step from idle to peak and back together, every training step, producing load swings of hundreds of megawatts on sub-second timescales. A separate mechanism can drop the entire load: protection response to a grid fault. In a 2024 Virginia event, ~1,500 MW of data-center load tripped off during a six-fault, 82-second sequence on a 230 kV line — enough that the surviving generation had to absorb the imbalance, and enough that NERC issued a rare Level 3 Essential Actions Alert and now treats large data centers as grid actors expected to ride through faults — an expectation the alert recommends but does not yet enforce, with a penalty-backed reliability standard still in development (NERC / Utility Dive, 2026).

Ride-through has become a goodput concern as much as a grid-interconnection one. A cluster that actually loses IT power, cooling or required state during a survivable grid disturbance converts that event into a full restart, paid in badput. A grid-visible transfer to backup while the IT keeps running incurs no such restart loss. The mitigation is the same transient-absorption stack that protects against the cluster's own load swings — per-GPU capacitance, rack BBUs, facility BESS, and NVIDIA's 400 J/GPU Intelligent Power Smoothing, which NVIDIA states can reduce peak current demand by up to 25% — now also tuned to keep the cluster online through utility-side faults rather than dropping load (NVIDIA / SemiAnalysis, 2025–2026). The choice is to engineer the facility to ride through grid disturbances — storage and smoothing capex, plus a regulator-facing study — and test delivered IT power, coolant response and recovery on the same event clock; otherwise a grid disturbance can become a cluster restart paid in goodput, while the utility still sees an unexplained load departure. Qualify the storage over that full envelope. → the full grid-interactive engineering — reactive support, frequency response, ride-through curves at the point of interconnection — is canonical in Chapter 4.10; the storage that backs it in Chapter 4.5.

None of this means the standards are wrong — it means they answer a question that is no longer the binding one. Uptime Tiers, TIA-942 Rated levels, and EN 50600 Availability Classes assess their defined infrastructure requirements; none certifies a numerical goodput guarantee. A Tier IV building tells a tenant the assessed power and cooling paths preserve the required service through the Tier IV fault conditions, with continuous cooling and separately stated outcomes for additional faults during maintenance; it says nothing about whether the cluster inside it loses 10% or 20% of its bought GPU-hours to badput the facility never sees. The standards remain the right tool for the question they answer — they govern the ceiling — but they cannot be the design-basis metric for a workload whose return lives in the gap below that ceiling.

The practical reconciliation is a two-tier design-basis: certify the facility to the availability class the workload's floor requires (higher continuity can pay for an unmasked interactive endpoint, while checkpointable training can price replay — both still owe the required states and customer obligations), then run a separate goodput design-basis that governs the silicon/storage/software redundancy the standards never touch. This is also where the redundancy primer in Chapter 0.5 gets extended: N, N+1, 2N and the distributed-redundant topologies are still the vocabulary, but you now apply them in two places — the facility power/cooling chain and the compute resilience stack — and the goodput model decides which application earns the spend. The standards landscape and topology selection are detailed in Chapter 12.1; the SLA that contracts goodput rather than availability is the subject of Chapter 12.4.

Put the mechanisms on one economic curve: incremental lifecycle cost on the x-axis, goodput or realized service value on the y-axis. For a training fleet whose largest avoidable losses are addressed first, the curve starts steep and flattens as those losses disappear; event frequency, recovery time and remaining shared causes set the actual slope and step order. The first dollars above the required facility-state floor go to the largest avoidable loss per dollar. Multi-tier checkpointing attacks wasted progress and slow recovery: Google’s June 16, 2025 report gives a 6.59% ML Goodput increase on a 35K-chip TPU v5p workload and checkpoint restore under one minute, not an end-to-end fleet MTTR guarantee. Spare pools and health checks rise when capacity waits dominate; cooling continuity, transient ride-through and the second facility path compete on the events they prevent. Price each rung against the loss still left after the previous purchase, or pay twice to recover the same GPU-hours. Serving fleets bend the curve differently: request retry and replica, zone or region capacity can mask a site event outright, so a single-site outage is recoverable at the fleet layer and 2N is not automatic; a single-site serving fleet has none of that, and for it the facility rungs climb back to the top. The right budget is where marginal goodput per dollar equalizes across the facility, the compute stack and the fleet — the point Chapter 12.5 locates for a named failure environment.

Choose recovery mechanisms that preserve the workload’s required state and complete inside its deadline, then price them against the same facility events. Reserve and storage expenditure earns its place through recovered work; underfunding the required physical states leaves a loss the software cannot mask. Carry the event-to-job mapping from Chapter 8.9 into the model, and carry the resulting measurements to the useful-output ledger in Chapter 14.1.