Cooling-Controls Transient Dynamics & Setpoint Stability

Chapter 4.5 told the electrical side of this story: a frontier training cluster is the worst load the grid has ever been asked to serve, because tens of thousands of synchronized GPUs ramp from idle to full power and back on millisecond timescales, and the power chain has to absorb the di/dt with stored energy before it propagates upstream. This chapter is the thermal twin of that problem. The same synchronized slam that stresses the busbar also lands, a few hundred milliseconds later, as a step change in heat flux at the cold plate — and a direct-to-chip loop has almost none of the thermal inertia that a legacy chilled-water plant used to ride through. What this chapter forces is a controls decision rather than a capacity one — Chapter 5.1 owns the capacity wall — namely how the controls behave in the seconds around a transient: how fast they may move, how they avoid oscillating, and what they do when the load disappears as fast as it arrived.

The chapter turns on four settings: slew-limit tuning, inertia-vs-response, dew-point margin, and capping coordination. Set any of them wrong and the cost is concrete — a throttled GPU (lost goodput), a hunting loop (mechanical wear and unstable die temperatures), or condensation on a cold manifold inside an energized rack (the failure mode that ends up in Appendix F). None of this is exotic; it is the ordinary consequence of running a low-inertia loop under a load profile that swings harder and faster than any cooling plant was historically asked to follow.

A legacy air-cooled hall was a giant thermal flywheel. Tens of thousands of cubic metres of room air, a raised-floor plenum, the thermal mass of the slab and the racks themselves, plus a chilled-water plant carrying hundreds of thousands of litres in buffer tanks and pipe volume — all of it stored enough sensible heat that a control loop could be sluggish and the room would barely notice a load step. A CRAH that took thirty seconds to ramp a fan was fine, because the air temperature could not move quickly in a volume that large. The plant rode through transients on stored thermal mass.

A direct-to-chip loop deletes that buffer. The coolant in contact with the die is a thin film in a microchannel cold plate; the technology-cooling-loop (TCS) volume serving an NVL72-class rack is on the order of ~200 L, and the transport time from cold plate to CDU heat exchanger and back is seconds, not minutes. The thermal resistance from junction to coolant is deliberately tiny — that is the entire point of liquid cooling — which means the die rides a small, power-dependent offset above the local coolant temperature. When GPU power steps by more than half of TDP in milliseconds (measured: power variations exceeding 50% of TDP within a few AC cycles; current swings from a 5–10 A idle baseline to 20–25 A in under 200 ms on a single inference node), it is that offset that jumps first — in milliseconds, buffered by the silicon and cold-plate mass — while the loop responds on the tens-of-seconds scale: dump a 115 kW step into the ~200 L inventory at maintained flow and the ideal fluid rise is only ~1.4 °C in ten seconds, before pipe and manifold metal even count. At maintained flow the first loop-level symptom is the return temperature climbing; supply stays wherever the CDU holds it. The transient that truly has no inertia is different: loss of flow. Stop the pumps and the thin film in the plate is all the coolant the die has — the junction reaches throttle in seconds. That one is a redundancy problem (Chapter 5.11), not a controls-tuning one, and conflating the two produces controls designed against the wrong time constant.

This is the inversion that catches operators migrating from air: in the old world, the cooling plant was slow and that was safe; in the DLC world, the cooling plant is the fast element in a tightly-coupled loop, and slowness is what gets you throttled. The GB200 loop is unforgiving precisely because the coupling is tight — ~170–235 L/min of secondary-loop flow per rack at the guide's ~7.5–12 °C cold-plate rise (vendor reference designs run a wider ~12 °C rise and so specify ~110–140 L/min for the same ~115 kW liquid share — flow and delta-T trade directly, so never quote one without the other), and a die temperature that sits a small, power-dependent offset above supply (the rack accepts supply up to ~45 °C; the closer you run to it, the less headroom you hold) — and a loop that lags a load step costs you throttled GPUs. The control system now lives in the goodput path.

The CDU is where the control decision physically happens. Its two actuators are the pump VFD (setting flow) and a modulating control valve — usually a three-way mixing valve on the facility-water side of the heat exchanger, which sets how much heat the TCS loop rejects and therefore the secondary supply temperature. Both actuators have slew limits: the maximum rate at which they are permitted to change. Vendor-typical envelopes for a predictive deployment sit around ±3% pump RPM per minute and ≤10% valve travel per minute, and a well-built CDU holds secondary supply within ±0.5 °C of setpoint in steady state.

The slew limit cuts both ways. Set it too slow and the loop falls behind a sustained load step: the return comes back hotter, the heat exchanger rejects less than the racks are producing, and over tens of seconds the accumulated energy drags secondary supply above setpoint until the die's headroom is spent and the GPUs throttle — lost goodput, the exact thing a frontier training run cannot afford. Set it too fast and you invite the opposite failure: the valve and pump overshoot the correction, the over-corrected temperature transports back to the sensor a few seconds later, the controller over-corrects the other way, and the loop hunts — a sustained oscillation in flow, temperature, and pressure that wears valve seats and pump bearings, fatigues quick-disconnects and hoses with pressure cycling, and swings die temperatures around the throttle threshold so the GPUs themselves oscillate between full and capped. Hunting is the cooling-controls failure mode catalogued in Appendix F, and it is almost always a tuning fault rather than a hardware one.

The reason this is hard is the transport lag in the callout above: any loop with significant dead time between actuation and feedback is prone to oscillation if its gain is too high. Classical fixes apply — lower proportional gain, lengthen the integral time, add a setpoint deadband so small excursions do not chase the actuator, and enable anti-windup so the integrator does not saturate while the valve is slew-limited at its travel cap. But the more durable answer in 2026 practice is to stop relying on feedback alone.

The three strategies are a spectrum, and real facilities blend them. The center of gravity in 2026 has moved decisively from the top row toward the middle: a purely reactive PID loop is fine for an inference hall with gentle load, but it is increasingly inadequate for a dense training cluster whose load steps are large, fast, and — critically — synchronized and predictable. A training step is a repeating waveform: the cluster slams to full power during compute, drops during a collective or a checkpoint, slams again. If the controls can see that waveform — via the GPU power-management telemetry, the scheduler, or even the BMC power signal — they can feedforward the valve and pump position so the coolant is already cold when the heat arrives, instead of chasing it. Predictive deployments that do this report holding die temperatures within ~2 °C of the throttle limit through the steepest spikes while cutting CDU energy 22–28% (because the loop stops over-pumping and over-cooling defensively). The bottom row — buy your way out with inertia — is the fallback when even feedforward cannot keep up, and it is where the cooling and electrical problems converge.

The cleanest way to make a cooling-controls problem tractable is to make the load step smaller before it ever reaches the cold plate — which is exactly what the chip-to-BBU-to-BESS spine of Chapter 4.5 does. Three coordinated levers blunt the slam:

GPU power-capping is the fastest and cheapest. The accelerator's own power-management firmware can clamp or ramp-limit board power, smoothing the di/dt at the source. Vendors have moved this on-die: NVIDIA's Rubin generation builds in rack-level power-smoothing on the order of a few hundred joules per GPU — MGX-rack capacitor buffers that flatten the worst transients before they reach the rack bus. A cap is also the graceful-degradation lever for cooling: when a thermal excursion is developing, throttling compute a few percent to hold die temperature is almost always cheaper than letting the loop chase it into instability or, worse, letting a chip trip. This is the goodput decision — a small, deliberate cap costs a sliver of throughput; an uncontrolled excursion costs a restart.

Battery-backed units (BBU) and facility BESS absorb the electrical transient so the grid sees a smoother draw. NVIDIA's production-BESS guidance for AI factories names transient absorption and power-smoothing as first-class BESS roles alongside ride-through, with closed-loop state-of-charge control sized to the workload's transient signature. Be precise about what that buys thermally: storage changes where the electrons come from, not how much heat the silicon makes — the rack's heat flux follows GPU power exactly, wherever the power is sourced, so only the power cap above actually flattens the thermal waveform. What the BESS buys the cooling plant is indirect but real: it keeps upstream electrical disturbances from becoming compute disturbances, so the thermal profile stays the workload's own.

The coordination decision is who arbitrates. If GPU capping, BBU smoothing, and CDU controls each react independently to the same slam, they can fight: the cap reduces heat just as the CDU finishes slewing colder, and now the loop overshoots cold (toward the dew-point floor, below). The mature pattern is a supervisory layer that knows the load forecast and sequences the responses — cap and smooth first, feedforward the cooling to the post-cap heat profile — rather than three reactive loops racing each other. → Chapter 4.5 owns the electrical-spine design; Chapter 14.7 owns the in-operation power-and-thermal management policy.

The dew-point floor interacts with the warm-water trend in an awkward way. Everything in Part 5 pushes coolant warmer — 30 °C-plus facility water to maximize free cooling and heat reuse (Chapter 5.1) — and warmer supply is comfortably above any realistic dew point, so condensation risk falls as supply temperature rises. Good. But warmer supply also means a smaller temperature margin between coolant and the die's throttle limit, which makes the loop less forgiving of a transient overshoot: there is less thermal headroom to absorb a momentary lag before the GPU throttles. So the warm-water decision is itself a transient-stability decision — it trades condensation risk down for throttle-margin risk up, and the control loop has to be tuned for the regime you actually run, near the middle of the operating band, not at a corner.

Hunting is the most common preventable failure in a commissioned liquid plant, and it is invisible at steady state — it only shows up under the synchronized-load conditions the workload actually creates. The mechanism is dead-time-driven oscillation: a loop with several seconds of transport delay between actuator and sensor will oscillate if its loop gain is high enough that a correction returns to the sensor still large enough to provoke an opposite correction. Three knobs and one architecture decision keep it stable.

• Gain and integral time. Lower proportional gain so a single correction does not overshoot; lengthen integral time so the controller does not wind up faster than the loop can respond. The classic symptom of too-aggressive tuning here is a regular sinusoidal swing in supply temperature with a period of roughly 4–6x the transport dead time.
• Deadband. A small setpoint deadband (e.g. ±0.3–0.5 °C) stops the loop from chasing sensor noise and micro-excursions, which is what drives high-frequency valve dithering and seat wear. The cost is a slightly looser steady-state band — an easy trade.
• Anti-windup. Because the actuators are slew-limited, the integrator will try to demand more correction than the valve can deliver during a fast transient. Without anti-windup the integrator saturates, and when the transient passes the loop dumps the accumulated error as a giant overshoot — straight toward the dew-point floor on a load drop. Anti-windup (clamping or back-calculation) is not optional on a slew-limited loop.
• Architecture: feedforward beats feedback for dead-time loops. No amount of feedback tuning beats the transport lag; the structural fix is to add a feedforward term from the load signal so the loop is mostly open-loop-correct and the PID only trims the residual. This is the same insight as the predictive-supervisor row in the table — it is the difference between chasing the load and leading it.

One more anti-pattern worth naming: controls fighting across layers. If the CDU PLC, the facility BMS, and a GPU-side thermal manager each run their own loop against overlapping setpoints, they interact like coupled oscillators and the whole system hunts even when each loop is individually stable. Decide which layer owns the secondary-supply setpoint and let the others observe, not actuate. This is the thermal analogue of the electrical arbitration problem above.

The operational consequence is that a cooling plant which passes a steady-state capacity test can still fail catastrophically under the workload's real transient profile, because the two are different physics. A Level 4 capacity proof says the plant can reject the rated heat at the rated flow indefinitely. It says nothing about whether the loop hunts when the load steps, whether the controls catch a slam before the die throttles, or whether the setpoint clamps above dew point on a fast drop. Those have to be demonstrated explicitly — which is why the integrated-systems-test (IST) for a liquid-cooled hall has to include transient cases, not just the load-bank-at-full sequence.

The transient acceptance set is short and specific: a synchronized step test (drive the load from low to full and back as fast as the load banks or a real workload allow, and confirm the controls catch it without throttling and without overshoot), an anti-hunting demonstration (hold the load at the worst-case operating point and confirm the loop settles inside its band without sustained oscillation), and a dew-point excursion test (drop the load fast and confirm the setpoint clamp holds secondary supply above the measured dew point throughout). Pair these with the metering and telemetry that make them observable in operation — the per-rack power, flow, supply-temperature, and dew-point signals are exactly the ones cooling controls need and the ones Chapter 4.12 meters. → cooling acceptance and CDU commissioning in Chapter 13.5; the full Level 5 IST and failure-mode demonstration in Chapter 13.6.