Component Failure Modes, Failure Rates & Fleet Reliability Data

This is the canonical home for one uncomfortable fact: a large AI training cluster must expect component interruptions during a run and recover from them. A traditional enterprise data center measures uptime in nines and counts annual outages on one hand. A 16,384-GPU training cluster experiences an unplanned interruption roughly every three hours — Meta's published Llama 3 405B run logged 419 of them over 54 days — and a well-run fleet still delivers over 90% effective training time through automation, not through preventing the failures (Meta, Llama 3 Herd, 2024). At this density and scale, the per-component physics guarantees components fail; the reliability problem is measuring the rate precisely enough to size spares, model availability, and detect the failures that hide.

This chapter establishes three things every other reliability chapter in the guide depends on. First, the failure taxonomy — hard, transient, and silent — which is the canonical fault vocabulary cross-referenced from the GPU operations view in Chapter 10.7, the redundancy view in Chapter 12.1, and the IST failure-demonstration view in Chapter 13.6. Second, the empirical fleet failure-rate data — the actual published numbers from at-scale operators, with their vintages and caveats, that you plug into an availability model rather than inventing. Third, the AFR modeling and burn-in discipline that turns raw component failure rates into a spares forecast and an acceptance gate. Every AFR in this chapter feeds the cluster availability and goodput roll-up in Chapter 12.5; the consolidated FMEA catalog these modes populate lives in Appendix F.

Every fault in a GPU fleet falls into one of three classes, and each demands completely different detection and recovery machinery; confusing them is a common operational error. The taxonomy organizes everything downstream: what you instrument, how fast you must react, and whether the failure is even visible at all.

Hard failures are the easy ones, paradoxically, because they announce themselves. A GPU throws an uncorrectable XID and falls off the PCIe bus, an optical transceiver goes dark, a CDU trips on low flow, a power supply faults. The component is unambiguously dead or unreachable; the job crashes or the node drops out; the telemetry screams. XID 79 ('GPU has fallen off the bus') is an archetypal hard-failure signal, but this guide does not assign it a portable fleet prevalence without a named equipment population, exposure window and study. These are expensive in lost goodput but cheap to detect: the recovery path is fail-fast, drain, restart from checkpoint, swap the FRU. Hard failures are where mature operators are already good, because the signal is loud.

Transient failures are the hard middle case. A correctable ECC error storm on HBM, a link that flaps rather than goes down, a thermal excursion that throttles a GPU for ninety seconds, an XID that self-clears on reset. The component is not dead — it works again after a power-cycle or a few minutes — but it degraded the run while it misbehaved, and it will very likely do so again. Transients are the raw material of lemon nodes: hardware that passes every point-in-time health check yet fails repeatedly under load. The decision a transient forces is not 'is it broken' but 'is it broken often enough to eject' — and getting that threshold wrong either keeps a repeat-offender poisoning runs or ejects healthy capacity. Correctable-ECC trends are one degradation signal to combine with XIDs, row-remap state, thermals, and load-correlated recurrence before deciding whether to eject hardware.

Silent failures are the dangerous class, because by definition nothing screams. Silent data corruption (SDC) is a computational error — a multiply that returns the wrong product, a memory read that flips a bit undetected by ECC — that produces no fault, no XID, no log line. The hardware reports success. The math is wrong. In training, an SDC silently corrupts gradients and weights; the loss curve drifts or diverges days later, and you cannot tell whether it is a bad hyperparameter, a data bug, or a single faulty multiplier in one of a hundred thousand chips. Meta’s 22 July 2025 reliability discussion reports about one SDC fault per thousand devices, without a stated exposure period. Its comparison with soft-error-related bit flips concerns a different error class, not a measured rise in one SDC rate. On a 100,000-device fleet, that statement cannot establish the fraction of defective machines or the probability of corruption during a multi-day run. Fund a deliberate detection program around the instructions, data patterns and operating conditions it covers; budget the GPU-hours for screening and the trusted progress lost before detection.

Fleet-level AFR is an aggregate that hides a strongly skewed distribution: a handful of components dominate the failure budget, and knowing which ones lets you target spares, burn-in, and detection where they pay off. The Meta Llama 3 root-cause breakdown is a useful public job-interruption dataset, but neither its counts nor its category mix is a portable equipment failure-rate distribution.

The GPU and its HBM dominate. In the Llama 3 paper's Table 5, 148 interruptions were attributed to faulty GPUs and 72 to HBM3 among 419 unplanned interruptions on one 16,384-H100, 54-day run. These are event counts, not unique failed devices or component exposure; the paper's printed percentages are internally inconsistent, so use the counts and preserve the named job boundary. This is not surprising once you see the physics: the accelerator package is the densest, hottest, highest-current component in the rack, and HBM stacks are the most thermally and mechanically stressed memory ever shipped at volume. Trend HBM health jointly with junction temperature, workload, ECC events, and remap history; as rack power rises, preserve thermal margin through the selected coolant-temperature, flow, and cold-plate envelope in Chapter 1.2.

Network and optics are the persistent long tail. Switches and cables accounted for 8.4% of Llama 3 interruptions, and link-flaps are as damaging as hard-down links because they corrupt collectives without obviously failing. At 800G XDR and the optics densities of a rail-optimized fabric, transceiver and cable failures scale with link count — a 100k-GPU cluster has millions of optical links, and even an excellent per-link AFR multiplies into a steady drip of fabric faults. The fabric is the failure domain that grows fastest as you scale out.

Infrastructure failures are rarer but higher-impact. Power and cooling faults are far less frequent than GPU faults per-event, but a single CDU trip or a PDU fault can take down an entire rack or pod at once — converting one component failure into dozens of simultaneous node losses. The Uptime data is stark: power is implicated in roughly 45% of impactful data-center outages (mostly UPS), and Uptime Institute's 2025 human-error survey findings use narrower denominators: 58% of human-error outages involved failure to follow an established procedure, while roughly 85% involved either that behavior or a flawed procedure. Neither figure is a share of all serious outages. The lesson for AI fleets is that the GPU dominates frequency while infrastructure dominates blast radius — and your FMEA in Appendix F must weight both.

Job interruption cadence belongs to a named source population and event definition, not to GPU count alone. Meta measured 419 unplanned interruptions over 54 days while training Llama 3 405B on 16,384 H100s — about one every three hours for that run, including hardware and software causes. A separate SemiAnalysis reference reports roughly seven days MTBF for one 512-H100 cluster at a top-tier operator. These are useful anchors, but they are not points on one arithmetic curve: fleet maturity, topology, job membership, correlated faults, software, detection policy, and the interruption denominator differ.

The stable scale lesson is directional: a larger synchronous job participates in more failure domains, so recovery and GOODPUT matter more as the job grows. Fit the effective job-level failure distribution from fleet telemetry, then compute checkpoint cadence from that measured distribution and the actual save/restart cost; do not infer a universal two-minute interval or a minute-level failure forecast from accelerator count. The checkpoint math is canonical in Chapter 9.4; operational tuning is in Chapter 14.4.

Because SDC by definition leaves no log line, detecting it is an active program, not a passive alarm — and the state of the art is a layered defense, with each layer trading coverage against the GPU-hours it steals from production. Meta's published CPU-fleet stack illustrates that tradeoff; it does not establish GPU-fleet coverage.

Fleetscanner is the offline sweep: dedicated CPU test patterns scheduled across Meta’s fleet on a reported 45–60-day CPU sweep cadence. It is the most thorough layer (over three years it reached ~93% coverage for a major defect family, with ~23% unique coverage no other method caught) but the most expensive, because the CPU under test is unavailable for its production workload. Ripple co-locates with live workloads, slipping millisecond-to-second test bursts into the gaps between real work, so it achieves fleet-wide coverage in days rather than weeks at near-zero opportunity cost — at lower per-pass depth. Hardware Sentinel is the newest layer: it watches application exceptions in kernel space and infers core-level SDC without allocating any test time at all, reporting effective CPU defect coverage roughly 1.74x over Fleetscanner and 1.92x over Ripple in the studied population (ASPLOS 2025); neither ratio is GPU/HBM coverage. The architectural lesson is that no single method suffices — you layer a deep-but-slow sweep, a fast-but-shallow in-workload probe, and an application-exception inference layer with monitoring and validation costs, and the union catches what any one misses.

For training specifically, the framework-level defenses matter as much as the fleet-level ones: redundant computation on a sample of operations, gradient/activation checksums, and divergence monitors that flag when a replica's numerics drift from its peers. These can catch covered SDC paths during computation; they do not detect every wrong result, and the CPU sweep’s 45-day clock is not a GPU-diagnostic delay estimate. The decision here mirrors the funding fork above — every layer you add costs GPU-hours or engineering, and the right depth is set by how catastrophic a silently-corrupted run is for your business. A frontier lab burning months of compute on one run has a strong reason to fund layered detection; a batch-inference shop can buy less only if output checks and replay contain silent errors before they escape. Neither workload can infer GPU coverage from Meta’s CPU results.

Component failure rates are not constant over life — they follow the classic bathtub: a high infant-mortality phase early, a low flat useful-life phase, and a rising wear-out phase late. Burn-in is the deliberate decision to pull infant mortality forward into a controlled acceptance window so it lands on a test harness instead of a customer's training run. It is a direct schedule-versus-reliability trade, settled at go-live.

The campaign is selected from the signed OEM/project/contract plan: declared stress profile and cycles, fleet exposure, measured discoveries by failure mode and node-hours, pass/re-soak dispositions, and a pre-agreed statistical stopping rule. Together AI and ClusterMAX described 72–168 hours in 2025 practitioner guidance; that is an example, not a universal bound. DCGM diagnostics can be components of the evidence package, but no run level alone is a universal acceptance gate. A separate October 2024 SemiAnalysis playbook recommends at least 3–4 weeks of factory high-temperature burn-in before deployment; it does not report a universal early-failure settling period. A burn-in campaign's claimed early-failure removal is only usable with its tested population, protocol, warranty interval and source locator; without those, treat the campaign as an acceptance gate, not a quantified reliability guarantee.

Burn-in does not end at acceptance — it transitions into a steady-state cadence. One possible day-2 schedule pairs a weekly deep node-health pass (dcgmi diag -r 3 plus NCCL on drained GPUs) with the continuous straggler/lemon detection that watches for the repeat-offenders burn-in could not catch; the actual cadence follows supported diagnostics, fleet evidence and the maintenance budget. A site may investigate a node running ~15% below a golden-reference benchmark as an illustrative trigger, after matching workload, power cap, temperature and software; automatic quarantine needs a validated threshold and false-positive budget, and the lemon-ejection decision — proven to cut 512+-GPU job failure rates from ~14% to ~4% and lift completion ~30% (Meta lemon-node studies, 2024) — is where transient-failure data becomes reclaimed goodput. Burn-in front-loads the cost of infant mortality; lemon ejection back-stops the transients that slip through. Both are detection programs paid for in GPU-hours, and both need the same accounting: compare reclaimed goodput with diagnostic GPU-hours, false quarantines and engineering cost in the 14.1 ledger.