Capacity, Power & Thermal Management in Operation

By the time a facility reaches steady-state operation, its single most expensive and least reversible decision — how many megawatts it can energize — is already made and paid for. The interconnection slot is won or lost, the transformers are humming, the cooling plant is sized. What remains is an operational discipline that the chip-bound era never had to learn: how to run a fixed power budget hot enough to earn its capital back, without running it so hot that a workload transient trips it. This is the day-2 face of the power-bound thesis. The constraint is no longer how many GPUs you can buy or even how many you can cool — it is how completely you can convert energized megawatts into goodput, request by request and step by step.

This chapter treats four operational functions as a single problem because they share one ceiling: live power management and oversubscription (running the budget full), transient management in operation (keeping the swings inside the budget), stranded-capacity and thermal operations (recovering megawatts the design left on the table), and capacity planning, workload-aware operations, and demand response (matching the ramp and the grid). The physics of the transients themselves is canonical in Chapter 4.5 (storage and ride-through) and Chapter 5.12 (cooling-controls dynamics); here we manage them operationally.

A facility provisioned for 100 MW of IT load rarely draws 100 MW. Nameplate is the sum of every device's worst-case rating; real workloads almost never align their peaks. The gap between nameplate and realized peak is power headroom, and the central operational question is whether to sell that headroom — to oversubscribe, deploying more racks than the budget would support at nameplate, betting that they will not all peak together. Get the bet right and you have manufactured capacity out of statistics: the canonical result is that an inference cluster can host roughly 30% more servers under the same power budget with negligible performance loss. Get it wrong and the budget overruns, the protection scheme fires, and you convert a statistical near-miss into a real outage.

The fork is not whether oversubscription is a good idea in the abstract — it is whether your workload leaves headroom to sell. Here the two archetypes diverge sharply. A synchronous training job runs every GPU in lockstep: the all-reduce barrier forces the whole cluster to draw peak power at the same instant, then idle together while gradients sync. That correlated swing leaves almost nothing to overbook — measured headroom on training clusters is on the order of ~3%. Inference is the opposite: thousands of independent requests arrive at random phases, their prefill (compute-bound, high-power) and decode (memory-bound, lower-power) stages uncorrelated across the fleet, so the aggregate draw is smooth and well below the sum of peaks — headroom on the order of ~21%. The same overbooking policy is free money on one hall and a feeder trip on the other.

The table is a posture ladder. As you descend from synchronous training to batch inference, two things move together: the workload's peaks become less correlated (more headroom to sell) and its tolerance for being throttled rises (a cheaper backstop). That is not a coincidence — it is why batch inference is the natural sink for oversubscription risk across a mixed fleet. The sophisticated operator does not pick one posture; it runs a heterogeneous budget where the interruptible, uncorrelated workloads absorb the overbooking that the synchronous, latency-bound workloads cannot. → the disaggregated RL picture is in Chapter 1.4; the inference burst profile in Chapter 1.3.

Oversubscription is a bet, and every bet needs a stop-loss. In a power budget that stop-loss is the power-cap hierarchy — a layered set of enforcement points that hold realized draw under the energized ceiling no matter what the workload does. The hierarchy spans four levels, each slower and broader than the one below it. At the GPU/node level, the accelerator's own power-management firmware clamps board power in microseconds by reducing clock frequency — the fastest and most surgical control. At the rack/PDU level, intelligent rack PDUs and the BMC enforce a rack budget across nodes. At the row/lineup level, the DCIM control plane allocates a shared budget across racks fed from a common busway or RPP. At the facility level, the EMS holds the campus draw under the interconnection limit and any demand-response obligation.

The fork here is where you set the binding cap and how fast it engages. Cap too low and you leave goodput on the table — GPUs throttled that the budget could have fed. Cap too high, or too slowly, and a correlated transient breaches the budget before the control loop catches it. The hard constraint is speed: a synchronous all-reduce drives the fleet from idle to peak in milliseconds, far faster than any facility-level EMS can react. That is why the binding, time-critical cap must live in the GPU firmware and the rack BMC — the only layers fast enough — with the DCIM and EMS layers setting budgets the fast layers enforce, not reacting to transients themselves. A facility EMS that tries to chase a training transient is always one control-loop period too late.

The power transient is what makes oversubscription dangerous in the first place. A synchronous cluster does not draw a steady load — it pulses. Compute phases pull every GPU to peak; communication and checkpoint phases let them fall together; a job start, a job kill, or a checkpoint reload steps the entire facility load in one swing. At fleet scale these swings are enormous and abrupt: NERC documented data-center load losses of roughly 1,500 MW on a single fault, with one Virginia event shedding ~1.5 GW in 82 seconds — events severe enough to trigger NERC's rare Level 3 alert and make grid-side ride-through a mandatory planning input rather than a courtesy. The physics of absorbing these swings — UPS, BESS, ride-through, and the controls that damp them — is engineered in Chapter 4.5 and Chapter 5.12. The operational question this chapter owns is narrower and continuous: how do you run the live facility so the transients you generate stay inside what your absorption layer and your grid contract can take?

The answer is a set of operational levers that shape the workload's power profile before it reaches the power chain. Power smoothing deliberately holds a floor under idle phases (a synthetic GPU load during the all-reduce trough) so the swing the grid sees is smaller than the swing the GPUs actually produce — trading a little energy for a lot of transient amplitude. Ramp-rate limiting staggers job starts and checkpoint reloads so a 100 MW cluster does not step its entire load in one second; the scheduler launches ranks in waves. Stagger and phase-offset scheduling spreads correlated peaks across time on purpose. And the storage layer — UPS and BESS — absorbs whatever residual swing the workload levers leave. The fork is how much of the transient you suppress in software (cheap energy, free capital, but it taxes goodput and is the operator's job) versus in hardware (BESS sized to the swing — capital up front, no goodput tax). Most facilities do both; the ratio is the design decision.

Stranded capacity is energized, paid-for power or cooling that cannot be converted into compute because some other resource caps out first. In a power-bound world that is capital spent on megawatts that earn nothing. It comes in three distinct strands, and the fork is which one you are actually suffering — because the recovery for each is different and confusing them wastes the recovery capital too.

• Cooling-limited. The power chain can feed the racks but the cooling plant cannot remove the heat, so racks sit half-populated or throttled. This is the dominant strand in halls built for air and asked to host liquid-class density — the cooling cliff showing up as a utilization ceiling. Recovery is mechanical: more heat-rejection capacity, warmer water, or a cooling retrofit. → Chapter 5.4.
• Power-chain-limited. Cooling has headroom but a transformer, busway, RPP, or breaker is the binding constraint — often because the design reserved fault-margin or redundancy capacity that nameplate planning never released to load. Recovery is electrical and partly statistical: oversubscription releases reserved headroom; rebalancing phases and circuits recovers fragmented capacity. → Chapter 4.6.
• Fragmentation-limited. Both power and cooling have aggregate headroom, but it is scattered — a kilowatt free here, two there — in pockets too small to land a 132 kW rack. This is the most insidious strand because the DCIM dashboard shows spare capacity that the placement engine cannot use. Recovery is operational: workload-aware placement, defragmentation campaigns, and consolidating partial rows.

Misdiagnose the strand and you spend in the wrong place. Add chillers to a fragmentation problem and you have bought cooling you still cannot fill; oversubscribe a cooling-limited hall and you trip the thermal protection you were already brushing against. The diagnostic discipline — read which resource is actually binding, per row, from telemetry before committing recovery capital — is the operational core of stranded-capacity management. → the telemetry that reveals the binding constraint is in Chapter 14.2.

Thermal operations is the cooling-side twin of power management: the continuous job of matching heat rejection to a load that swings as violently as the power draw does. The same all-reduce that pulses the power chain pulses the heat load — and the cooling plant's thermal mass and control loops respond on a slower timescale than the electrical ones, which is its own hazard. A GB200-class rack with ~1 kW+ GPUs can thermal-trip within seconds of a cooling-loop loss; there is no air buffer at liquid density to ride through on. That makes UPS-backed coolant pumps and N+1/2N heat rejection not redundancy luxuries but operational survival requirements.

The day-2 levers are setpoint and flow management against the DLC envelope: holding coolant inlet inside the ~20–25 °C window, flow at the rated ~1.2–2.0 L/min per kW, and the secondary supply above white-space dew point to stay 100% sensible. The fork in steady state is how warm you run the loop. Warmer water expands free-cooling hours and crushes the cooling share of PUE (warm-water DLC plants approach ~1.1 versus 1.3–1.5+ for air), but it shrinks the thermal margin to the throttle threshold — a GB200 that deviates from inlet spec can lose up to ~50% of its clocks. Run cold and you waste compressor energy buying margin you may not need; run warm and you are operating closer to the cliff, where a transient that would have been absorbed becomes a throttle event and a goodput loss. Cooling-controls stability under these transients is canonical in Chapter 5.12; the piping and CDU mechanics in Chapter 5.13; predictive detection of plant degradation (cooling-capacity loss 14–30 days out, chiller-bearing wear 4–8 weeks out) in Chapter 14.5.

Capacity planning in a power-bound facility is the discipline of never letting demand for megawatts outrun the supply you have energized, while never leaving so much energized capacity idle that the capital decays unused. It is a forecasting problem with an asymmetric loss function: under-provision and you turn away revenue against a depreciation clock that runs whether the racks are full or not; over-provision and you have stranded energized megawatts, the costliest resource on the site. The forecast that matters is not steady-state load — it is the density ramp. The substrate must be planned against the generation you will deploy next, because the irreversible parts (transformer capacity, busway ampacity, cooling-plant tonnage, floor loading) cannot be retrofitted on the timescale that rack density is climbing: 40 kW air-cooled racks gave way to ~132 kW NVL72, heading to ~190–230 kW Rubin-class and ~600 kW Kyber-class. A hall whose power and cooling were sized to today's density strands its own future.

The operational fork is the reserved-capacity buffer: how much energized headroom you hold back against the next ramp step and against failures. Hold too little and the next GPU generation arrives with nowhere to land — the density-ramp trap, now an operational reality rather than a scoping one. Hold too much and you are paying to idle megawatts. The discipline is to forecast the ramp curve generation-by-generation, size the irreversible substrate to it, and keep the reversible IT fit-out matched to current draw — reserving the headroom you cannot retrofit (floor, water, ampacity) while deferring the spend you can. The interconnection and energization sequencing that feeds this forecast (50–100 MW tranches, energized capacity leading commissioned load) is in Chapter 3.2; the scoping-time version of the ramp decision is in Chapter 1.1.

The most powerful lever an AI factory has over its own power profile is the scheduler. Because a large share of the fleet's load is interruptible (batch inference) or schedulable (RL rollouts, off-peak training), the operator can shape when and where power is drawn — not just react to it. Workload-aware operations is the practice of treating the power budget and the job queue as one optimization: placing interruptible work where stranded capacity hides, throttling low-SLO tenants first when a cap engages, smoothing the aggregate profile by phase-offsetting correlated jobs, and shifting deferrable load to cheap or low-carbon hours.

That same flexibility is what makes demand response and curtailment economically real for AI loads — and what makes it dangerous. The grid will pay handsomely for flexible load: studies find roughly 98–100 GW of US load integratable at just 0.5% annual curtailment, with a 1–2% peak-demand cut translating into 0.5–2.8% rate reductions system-wide. For a facility, participating means accepting curtailment events (averaging ~2 hours) in exchange for a better interconnection deal or a demand-response payment — increasingly a condition of large-load tariffs, not an option. The fork is which workloads you make eligible. Curtail batch inference or defer a checkpointable training run and the cost is near zero — exactly the load demand response was designed for. Curtail an online-inference SLA or a synchronous run mid-step and you destroy goodput value that dwarfs the demand-response payment: a breached latency SLA, or a forced checkpoint-and-restart that throws away hours of correlated compute. The revenue is real, but it must be earned from the interruptible tail of the fleet, never from its latency-bound or synchronous core.

The final function folds power, thermal, and workload management into a single objective: useful work per megawatt-hour. PUE measures only the facility overhead — and at warm-water DLC's ~1.1 there is little overhead left to chase. The frontier has moved inside the white space, to metrics like work-per-watt and goodput-per-MWh that count whether the IT load itself is doing useful computation or burning power on idle GPUs, retries, throttled clocks, and failed jobs. A facility at PUE 1.1 running GPUs at 40% MFU is wasting far more energy than one at PUE 1.2 running at 90%. Energy operations therefore reaches past the cooling plant into the cluster: raising MFU, cutting failure-induced rework, and timing flexible load to low-carbon, low-cost grid hours. The efficiency levers that matter most in 2026 are no longer mechanical; they are scheduling and reliability levers. → the full KPI and reliability-economics treatment is in Chapter 14.1.