Fleet Reliability, Fault Tolerance & Autonomous Recovery

A larger synchronous job participates in more component and service failure domains, so job interruption becomes a first-order goodput constraint. Meta reported about 47.7 days MTTF for an 8-GPU job and 7.9 hours for a 1,024-GPU job in its named research-cluster population. Those observations support the direction of scale, but they do not justify a universal six-figure-GPU cadence without fleet telemetry and explicit independence/common-mode, job-membership, software, and event-definition assumptions. That is why fleet reliability is an engineering discipline of its own rather than a facility-uptime line item.

This chapter is about the system that turns that failure rate from a goodput catastrophe into a manageable tax. It is built from three moving parts that must be designed together: the detection-to-recovery loop (how fast you notice, isolate, and restart), the fault-tolerance strategy (hot spares versus elastic/redundant training versus algorithmic resilience), and the checkpoint/restore substrate that decides how much work a failure erases. Every reliability dollar can go to facility nines, to recovery speed, or to spare capacity, and the three buy very different amounts of goodput. We name the canonical homes for the supporting math — checkpoint intervals in Chapter 9.4, the failure taxonomy and fleet AFR data in Chapter 14.3, and the availability-vs-goodput rethink in Chapter 12.2 — and concentrate here on the operational loop that ties them together.

Two facts collide to produce the modern training-reliability problem. First, synchronous coupling makes every node a single point of failure: in a data-/tensor-/pipeline-parallel run the job advances at the speed of its slowest rank, and a dead rank halts all of them. Second, the number of participating failure domains grows with job size. Under an explicit independent-identical-rate model the aggregate hazard grows with membership, but real fleets also carry correlated hardware, shared-service, network, software, and detection effects. The operational input is therefore the measured effective job-level interruption distribution, not a universal hourly cadence.

Several public empirical anchors are useful but not directly interchangeable. Meta's Llama 3 405B run logged 419 unplanned interruptions over 54 days on 16,384 H100s — about one every three hours — of which approximately 78% were hardware-caused; the paper reports 58.7% as GPU issues although Table 5's counts and percentages do not reconcile, yet the team still achieved over 90% effective training time in that Llama 3 campaign through aggressive automation and only three manual interventions. SemiAnalysis's October 2024 operator playbook reports about 7 days MTBF for one 512-H100 cluster at a top-tier operator and separately recommends at least 3–4 weeks of factory high-temperature burn-in before deployment. The first is not a per-GPU rate; the second is not an observed universal settling time. Alibaba's Unicron production study found a 43.4% failure rate among the top 5% most resource-intensive large jobs, about 37% hardware-attributed and roughly 73% recoverable via restart. At scale, failure is the steady state to engineer for.

The consequence for design is that a facility-topology classification does not measure training output. Uptime Tier outcomes describe topology under defined conditions; a 100k-GPU synchronous job loses far more useful time to internal hardware faults that the facility's 2N power and cooling do nothing to prevent. The metric that governs the return on a training cluster is goodput (equivalently ETTR, effective-training-time ratio): productive GPU-time divided by wall-clock GPU-time. This is the canonical pivot of Chapter 12.2; here it is the lens through which every recovery decision is scored.

Every interruption runs through the same five-stage loop, and the time spent in each stage is what you actually control. Detect the fault; drain the affected node or rack out of the job; diagnose the root cause; remediate (reboot, reseat, RMA, or replace); and restart the job from the last good checkpoint onto healthy hardware. The cluster's goodput is set by how fast this loop closes and how often it has to run. The failure taxonomy that the loop must classify — hard faults, transient faults, and silent data corruption — is canonical in Chapter 14.3; here we treat detection as a given input and focus on the loop's economics.

The non-obvious lever is that detection latency and restart latency dominate, not repair latency. Repair (an RMA, a reseat) happens asynchronously on a drained node while the job runs on a spare; it is off the critical path. What sits on the critical path is the time to notice the fault (a hung collective can stall a job for minutes before a watchdog fires) plus the time to load a checkpoint and re-establish the fabric. This is why the reliability spend with the largest return is rarely 'better hardware': it is faster watchdogs, faster checkpoint loading, and a warm spare ready to slot in. Multi-tier checkpointing has driven restart from the legacy 15–30 minutes down toward under two minutes, and that single change can move goodput by several points at frontier scale.

The table is a budget allocator. The three rows marked 'on critical path' — detect, drain, restart — are where wall-clock goodput is won or lost; the two marked off-path can be slow and asynchronous after safe fencing and replacement placement, provided you have compatible spare capacity to keep the job running while they complete. This is the structural argument for hot spares: they convert remediate from an on-path stall into an off-path background task. It is also the argument for multi-tier checkpointing: it attacks restart, a potentially expensive on-path stage, by keeping a recent checkpoint in node-local memory or NVMe provided that copy survives the named failure domain and contains validated recoverable state. The checkpoint cadence and tiering math that governs how much a failure erases is canonical in Chapter 9.4.

At a fleet failing more than once an hour, a human-in-the-loop recovery process is a bottleneck — the operator becomes the MTTR. The 2025–2026 answer is to make the loop autonomous: a fleet control plane that detects a drained or unhealthy node, runs automated triage to classify the fault, attempts remediation (power-cycle, reflash, re-test) without a ticket, and only escalates to a human when it cannot resolve the fault itself. NVIDIA's Mission Control packages this for GB200/GB300 NVL72 as three coupled components — autonomous job recovery, autonomous hardware recovery, and the NVIDIA Resiliency Extension (NVRx) — running automated health checks at the tray, rack, and system level and executing break-fix workflows that open support tickets only for what cannot auto-resolve. Hyperscalers run their own equivalents (Meta's automation took Llama 3 to over 90% effective training time with only three manual interventions across 54 days), and neocloud operators differentiate on the maturity of exactly this loop.

The decision here is not whether to automate detection — everyone does — but how much authority to grant the loop to act. Full autonomy (auto-drain and auto-restart the job on a confidence threshold) maximizes goodput but can amplify a fault: a mis-classifying triage routine can eject healthy nodes, a restart storm can thrash the scheduler, and a shared firmware bug can be 'remediated' onto every node in turn. The guardrails that make autonomy safe are the same ones that make any control loop safe — rate limits on automated actions, quarantine of repeat offenders (lemon-node ejection), and a circuit breaker that hands control to a human when the action rate or failure rate spikes. An under-automated loop loses goodput to human latency; an over-automated loop without guardrails loses goodput to self-inflicted instability.

Once detection and recovery are fast, the next fork is how the job survives the failure — and there are three families, each trading capacity, framework coupling, and recovery speed differently. The choice determines both the spare-capacity tax you pay and how deeply reliability is wired into the training stack.

Hot spares + fast restart is the operationally simplest and most common posture: hold a pool of healthy GPUs idle (typically a few percent of the fleet), and on failure drain the bad node, slot in a spare, and restart the job from the last checkpoint. The cost is the idle spare capacity and the full restart latency; the virtue is that it is framework-agnostic and easy to reason about. Elastic / redundant training lets the job continue on fewer nodes (shrinking the world size and re-sharding) or run with redundant replicas so a single failure does not require a full restart — eliminating the spare tax and much of the restart stall, at the price of coupling reliability tightly to the training framework (the scheduler, the parallelism plan, and the collective library must all cooperate to reshard live). Algorithmic fault tolerance goes further into the math: techniques like nonuniform/elastic tensor parallelism reduce the goodput amplification a single GPU failure causes, and redundant-computation or erasure-coded schemes let the job tolerate a fault without re-execution — the lowest overhead in principle, but the least mature and the most workload-specific. A fourth family emerged commercially in 2026: live GPU migration (Clockwork TorchPass), which sits between the framework and the scheduler and migrates a running job onto a spare without a restart — vendor-benchmarked at zero lost steps on a 64-H200 failure injection versus 869 for checkpoint-restart, and independently tested by SemiAnalysis, which rated it the fastest fault-tolerant approach it measured. It is licensed software, the spare pool is still required, and the hard-failure path degrades toward just-in-time checkpointing — so treat it as a restart-latency eliminator for planned and XID-class faults, not a repeal of the checkpoint discipline. These map onto the operational reliability-engineering treatment in Chapter 14.4.

Moving down this table changes operational complexity, reserve capacity and recovery speed; elastic or redundant training also pays in framework coupling. Hot spares are something an operations team can run with a stock framework; elastic and redundant training require the training stack itself to be reliability-aware, which is why they are most common at the labs that own their framework end-to-end. The right answer is a function of how scarce your spare capacity is (a power-bound fleet must price every idle reserved GPU), how much a stall costs (a contracted run with a deadline cannot tolerate restart storms), and how much control you have over the training stack. A hybrid can use hot spares first and qualified elastic shrink when they are exhausted. A spare must match memory, tuple and fabric locality; free inventory in the wrong domain cannot back the promise. Test simultaneous claims on shared reserve before selling it twice.

The reliability budget has three competing claims — raise MTBF (better hardware, more burn-in), shrink MTTR (faster detect/restart, hot spares), or add facility nines (2N power, redundant cooling) — and at frontier scale they are not equally productive. The goodput of a checkpointable job is, to first order, uptime fraction = MTBF / (MTBF + MTTR + lost-work-per-failure), where lost work is modeled from the interval between checkpoints durably committed inside the declared failure domain; the RPO boundary is the newest checkpoint recoverable in that domain. This is a failure-loss illustration rather than a full goodput accounting: it carries no term for the ordinary cost of checkpointing a healthy job, so it tends to 100% in the no-failure limit even where every 100 s of useful training pays 10 s of blocking save — a true 90.9%. Take the complete accounting — productive work, ordinary saves, detection and recovery, recomputation — from Chapter 9.4. Because MTBF falls as you scale and is hard to move (the hardware is what it is), the dominant levers are MTTR (recovery speed) and lost-work-per-failure (checkpoint cadence) — both of which are software and control-plane investments, not facility ones.

At a named fleet’s measured failure distribution, checkpointing and hot spares may dominate the next-dollar goodput comparison because facility topology does not prevent silicon, software, or membership failures. That does not make 2N categorically wasteful: it prevents only the modeled facility states, whose frequency, interruption, post-event load, recovery consequence, common modes, and contractual value must be compared with compute-stack resilience. The Young/Daly checkpoint derivation is in Chapter 9.4; the state-based economic comparison is in Chapter 12.5.

Everything above is the training story, where one fault stalls one tightly-coupled job. An online-inference fleet inverts almost every assumption and therefore inverts the reliability strategy. At the API, inference requests remain retry-oriented, but the infrastructure has two serving tiers: in independently replicated serving, a node failure drops one replica's in-flight requests; in frontier serving, one replica can span an NVL72 scale-up domain, so a node or fabric failure can remove the distributed replica and every request it is serving. There is no training checkpoint to restore — the unit of failure is a request at the API and a replica or domain in the infrastructure. The reliability targets are correspondingly different: request success rate and SLO attainment (TTFT/TPOT within budget) rather than goodput-as-effective-training-time. Required maintenance and fault outcomes select the facility topology, not the inference label; revenue-critical serving often justifies a stronger posture than checkpointable batch work.

So the fault-tolerance toolkit shifts. For inference you invest in fast health-checking and load-balancer ejection (pull a sick replica out of rotation in seconds) and, for distributed replicas, domain-aware health, replica-level draining, and whole-domain spare/placement policy, geographic and zonal redundancy (so a rack, hall, or region failure degrades capacity rather than availability), and graceful degradation (shed or queue low-priority traffic, fall back to a smaller model, preserve the SLO for what remains) rather than checkpoint cadence and hot training spares. KV-cache recovery cost follows context length and replica design: short prompts are cheap to recompute, while long-context prefill can make recomputation or replication a material capacity and latency decision. Inference recovery therefore couples API retry with replica/domain recovery — and prefix-cache locality (Chapter 9.7) means losing a replica still costs cache warmth and latency. The serving-engine and SLO mechanics that this reliability posture protects are owned in Chapter 10.11; the multi-region failover and DR design in Chapter 12.3; and the SLA/goodput-contract framing in Chapter 12.4.

The same reliability mistakes recur, because each comes from optimizing one number in isolation rather than goodput end-to-end:

• Choosing topology from checkpointability. Checkpoint-and-resume lowers the consequence of some interruptions but does not prove that N/N+1 is sufficient or that 2N is waste. Compare the named facility states, interruption, post-event loading, recovery, independence, common modes, and contract with checkpointing and spares (Chapter 12.5).
• Optimizing MTBF instead of MTTR. Pouring the reliability budget into hardware screening to push MTBF up a few percent, while restart still takes twenty minutes. At a fleet failing every few hours, compare the loss avoided by halving MTTR with that avoided by fewer interruptions, including lost progress and reserve. The larger return depends on the measured clocks and the intervention cost, rather than a universal preference for software.
• An autonomous loop without lemon tracking or rate limits. Auto-rebooting and re-admitting nodes with no health history and no remediation rate limiter, so a single lemon flaps the job and a correlated firmware fault triggers a fleet-wide remediation storm.
• Treating SDC as someone else's problem. Relying on the hang-and-restart loop to catch corruption it is structurally blind to, and discovering days of poisoned training only at the next eval. SDC needs a dedicated detection program (Chapter 14.3), not a louder watchdog.

Buy recovery time from the stage that dominates the measured critical path, with its spare, checkpoint and ownership dependencies inside the test. Faster storage earns nothing while allocation is blocked; more spares earn nothing when the checkpoint cannot resume valid work. Charge the remedy against 14.1, using 9.4 for checkpoint semantics.