Operational KPIs, Goodput & the Reliability Economics of AI Factories

Parts 1 through 13 of this guide build the machine. This part operates it — and operating it is where most of the lifetime money is won or lost, because the asset depreciates whether or not it is producing. The reflex inherited from twenty years of enterprise data-center operations is to manage the facility against availability: nines of uptime, hours of downtime per year, Tier classes. That reflex is not wrong so much as it measures the wrong thing. An AI factory is a depreciating compute asset whose return is set by how many accelerator-hours land on the revenue-bearing critical path. A hall can be 99.995% 'available' and still throw away a quarter of its compute to stragglers, restarts, silent corruption, and idle hot spares — and the income statement will not care that the UPS never dropped.

This chapter is the framework for Part 14: it reframes the operating objective from availability to goodput, defines the goodput stack (ETTR, ML goodput, MFU/MBU) at engineering depth, characterizes the AI failure environment that makes day-2 different from traditional IT, sets out the blast-radius problem that turns one component fault into a cluster-wide stall, and assembles the operations scorecard that the rest of Part 14 instruments. The deep treatment of why goodput beats availability as a design target lives in Chapter 12.2; this chapter is the operational counterpart — how you measure it, what it costs to move it, and what the scorecard looks like once the building is live.

The first day-2 decision is which number sits at the top of your operations dashboard, because it silently re-weights every reliability investment below it. Availability asks: what fraction of wallclock time is the facility energized and the IT reachable? It is a facility-centric, binary-per-component view inherited from the Uptime Institute Tier model — Tier III at ~99.982% (about 1.6 hours of downtime per year), Tier IV at ~99.995% (about 26 minutes per year). Goodput asks a different and harder question: what fraction of the accelerator-hours you paid for actually advanced a job on the critical path? The two diverge sharply for AI, and that divergence is what Part 14 instruments.

Consider a synchronous pre-training run on 16,384 GPUs. The facility never loses power; by the availability definition it is at five-nines. But a single GPU fails roughly every three hours at that scale (Meta's Llama 3 405B snapshot), and each failure idles all 16,384 accelerators until the job restarts from its last checkpoint. The 'available' facility is hemorrhaging goodput through a mechanism the availability metric cannot see, because the failure domain is the job, not the component. This is why the AI-cluster reliability rethink (Chapter 12.2) argues that for checkpointable training, spending on 2N facility power to chase nines is often capital misallocated — the same dollars buy far more goodput as faster checkpointing, hot spares, and recovery automation. The choice determines whether your next reliability dollar goes into a redundant feeder or into a flash-checkpoint tier.

'Goodput' is not one number; it is a stack of nested ratios, each measuring a different kind of waste, and conflating them is the most common source of dishonest operating reporting. Read from the outside in, each layer multiplies the one below it, and the product is what the dollars actually buy.

ETTR (Effective Training Time Ratio) is the outermost layer and the cleanest measure of cluster reliability: the ratio of productive runtime to the available wallclock time of a job run, ranging from 0 to 1, accounting for queueing delay, restart overhead, and re-computation of lost progress (Meta, Revisiting Reliability in Large-Scale ML Research Clusters, 2024–25). ETTR is model-agnostic — it does not care what fraction of a GPU's FLOPs a kernel extracts — which is exactly why it is the right SLA target between an operator and a training customer. Meta's largest jobs (>1024 GPUs) sustained average ETTR above 0.9 with one-hour checkpoint intervals on shared, congested clusters; pushing a hypothetical 100k-GPU run to 0.9 ETTR requires driving checkpoint and restart overhead down to roughly two minutes each.

ML goodput (Google's formulation) is the aggregate productive work completed per unit time across the cluster, with all non-productive time accounted as 'badput': program startup, data-loading stalls, checkpoint writes that do not overlap compute, failed steps, wasted progress since the last checkpoint, and scheduling gaps. Where ETTR is per-job, ML goodput is the fleet-level rollup, and it is the number the operations scorecard reports. The industry average sits near 90%; best-in-class operators market ~96% (SemiAnalysis ClusterMAX / CoreWeave). That six-point gap is not rounding — at a 1 GW factory it is the difference between roughly 90 MW and 40 MW of accelerator power doing nothing useful.

MFU (Model FLOPs Utilization) and MBU (Model Bandwidth Utilization) are the innermost layer — they measure efficiency while a job is running, independent of failures. MFU is the fraction of the hardware's peak FLOPs the model actually realizes (compute-bound training; commonly 30–50%, >50% considered strong on Hopper). MBU is the analogous ratio for memory bandwidth (the binding constraint on autoregressive inference decode). A job can have a 0.95 ETTR and a 35% MFU — perfectly reliable, half-idle on the math — which is why the three metrics must be reported together. Realized output = ETTR x MFU/MBU x peak, and an operator who quotes only one of the three is hiding the other two. → metric definitions in Chapter 0.3.

The table is a hierarchy of denominators, and the operational sin is comparing two facilities on different rows. A neocloud quoting '99.99% uptime' is on the bottom row; a customer who actually loses 20% of their training run to stragglers and restarts is living on the top row. The contractual maturity of 2026 is that goodput, not uptime, is becoming the acceptance and SLA basis — operators gate production handoff on sustained NCCL/collective performance and low error rates over multi-day soaks, not power-on uptime, because each failed GPU can cost 15–20% performance and a mid-run failure destroys training economics. The industry is converging, slowly, on a contractual definition of 'goodput acceptance' (sustained collective bandwidth, an error budget, a soak window) that buyers can hold integrators to. → acceptance and IST in Chapter 13.6; goodput-vs-availability theory in Chapter 12.2.

Traditional enterprise IT operates a fleet of loosely-coupled, independently-failing servers: one box dies, a load balancer routes around it, and the blast radius is one request. The AI factory inverts every assumption behind that model, and the day-2 reliability program has to be rebuilt from the inverted premises.

The components fail far more often. Meta's research clusters log 6.50 failures per thousand node-days (RSC-1) and 2.34 per thousand node-days (RSC-2) — roughly 5x10⁻³ failures per GPU node-day. That sounds small until you multiply by scale: an 8-GPU job has a mean time to failure of ~47.7 days, but the observed cluster MTTF drops roughly in proportion to GPU count — to single-digit hours around 1,024 GPUs (~7.9 hr), to a couple of hours at 16,384 GPUs (~1.8 hr, Meta-observed and inflated by software-induced failures), and to minutes at the ~131,072-GPU scale. The intermediate figures are empirical, not a strict arithmetic chain, so they do not divide cleanly from the single-node base rate. The takeaway holds regardless: failure rate rises with GPU count, so the largest jobs spend a meaningful fraction of their life recovering rather than computing. This is the arithmetic that makes recovery speed, not component MTBF, the dominant goodput lever at frontier scale. → fleet failure-rate data in Chapter 14.3; operational recovery in Chapter 14.4.

The failures are not all loud. The AI fleet has three failure classes, and the dangerous one is invisible. Hard failures (a GPU falls off the bus, a link drops) announce themselves and trigger a restart. Transient failures (a correctable ECC storm, a thermal throttle) degrade goodput without stopping the job. Silent data corruption (SDC) is the third and worst: a marginal device computes a wrong result with no error flag, quietly poisoning gradients or activations. Meta finds roughly 1 in 1,000 machines affected by SDC, and for large training runs an SDC event is expected every one to two weeks; Google estimates an SDC event every week or two during Gemini training. Soft-error rates have worsened with process shrinks — from roughly one failure per year at 65 nm to one per ~1.5 hours at 16 nm — so this is structurally getting harder, not easier. Detection is now a standing fleet program (Meta's Fleetscanner and Ripple run ~2.5 billion test seeds per month). → SDC mechanisms and detection in Chapter 14.3.

The defining structural feature of the AI failure environment is that the failure domain is not the failed component. In a tightly-coupled synchronous job, one GPU stalling on an all-reduce stalls the entire collective, and the whole job moves at the speed of its slowest straggler. The scale-up fabric amplifies this further: a failed NVSwitch tray degrades bandwidth for all 72 GPUs in an NVL72 domain, and a tensor-parallel group of 64 GPUs with 0.1% of GPUs failed drops to roughly 94% effective availability for the group. The blast radius of a single fault is the size of the coupling domain you chose at design time, so domain sizing is a reliability decision as much as a performance one.

The cascade goes wider than the job. Meta found that 16% of total failure-related goodput loss came from secondary preemptions — small jobs getting evicted to free resources for a large job's restart. One large-job failure ripples into idle time across unrelated workloads sharing the cluster. This is the day-2 reason operators agonize over the NVL72-vs-NVL36x2 fork (one 120 kW domain vs two 66 kW domains): the larger domain lifts the tensor-/expert-parallel ceiling and the achievable MFU, but it doubles the blast radius of a single tray or cold-plate fault. There is no free choice here — you are trading peak efficiency against failure containment, and the right answer depends on whether you manage against MFU or against goodput. → scale-up domain sizing in Chapter 8.5; lemon-node ejection in Chapter 14.4.

Day-2 reliability is not a cost to minimize; it is an investment to optimize against a measurable return, and the return is denominated in goodput. Reframe the spend with the canonical economic anchor: a 1 GW AI factory carries roughly $8.5B/yr in all-in TCO (Epoch AI), and at the short, frontier-economic depreciation life the figure runs higher still. If you are throwing away the difference between 96% and 90% goodput, you are wasting roughly six percentage points of an $8.5B/yr asset — on the order of $500M/yr of compute producing nothing. Against that denominator, the reliability program — fast checkpointing, hot spares, health-checking, SDC scanning, lemon-node ejection — is one of the highest-return investments in the entire building, and the industry treats reliability overhead as a deliberate 6–21% slice of TCO precisely because the alternative is more expensive.

The economics also explain the hidden-tax structure that ClusterMAX exposed: holding GPU sticker price constant, a gold-tier neocloud's total cost runs lower than a silver-tier provider's by 5–15% on large training workloads, and a hyperscaler's 36-month total swelled to 1.10x a gold-tier neocloud's — a 10% hidden tax — entirely through goodput loss (10.53% vs 6.14%) that never appears on the rate card. The cheapest GPU-hour on paper can cost the most once delivered, because reliability is priced in goodput, not in the quoted $/GPU-hr. For fault-tolerant workloads (single-node inference), the gap collapses toward zero — which is exactly why the metric fork at the top of this chapter is not optional: the value of a goodput point is workload-dependent, and so is the reliability spend that is rational to chase it. → unit economics in Chapter 1.8.

An operations program is only as honest as its scorecard, and the scorecard is only as honest as its denominators. The day-2 KPI set spans three layers — the workload, the fleet, and the facility — and the discipline is to report all three with named denominators rather than collapsing them into a single flattering 'uptime' figure. The workload layer is goodput and ETTR (and MFU/MBU underneath); the fleet layer is failure rate per node-day, MTTR decomposition, lemon-node rate, and SDC detection rate; the facility layer is availability, PUE/WUE, and power/thermal headroom. Each layer has a different audience: the customer cares about goodput, the reliability engineer cares about the fleet layer, and the facility team and lenders care about the bottom layer.

The recurring day-2 failure is a scorecard that reports the bottom layer (because it is the one traditional DCIM measures well) and is silent on the top layer (because it requires IT/facility telemetry correlation the legacy stack never had to do). A facility can show a perfect availability scorecard while its customers are quietly losing a fifth of their compute to badput the operator never instrumented. Closing that gap — wiring the facility telemetry to the workload telemetry so that a thermal event, a power transient, and a training stall can be correlated to a single root cause — is the central task of the observability chapter that follows. → telemetry and IT/facility correlation in Chapter 14.2.

The same day-2 mistakes recur, each one a consequence of reaching for the facility-availability worldview when the workload obeys goodput economics. Three are worth naming:

• Buying nines a checkpointable job does not value. Commissioning 2N / Tier-IV power for a synchronous training fleet that already tolerates checkpoint-and-resume. The capital would return far more as goodput — faster checkpointing, hot spares, more GPUs — than as facility availability the workload cannot monetize. → Chapter 12.2.
• Reporting availability and calling it goodput. A green uptime dashboard over a fleet quietly losing 15–20% of its compute to stragglers, restarts, and badput. The denominator is wallclock-of-the-facility, not accelerator-hours-on-the-critical-path, and the gap is invisible until a customer measures their own ETTR and disputes the SLA.
• No standing SDC program. Treating silent corruption as a rare anomaly rather than a continuous fleet condition (~1 in 1,000 machines, an event every 1–2 weeks at scale). Without continuous scanning and checkpoint validation, multi-day rollbacks of poisoned runs are the ones the operator is structurally blind to.