CDUs & the Secondary Loop

Direct-to-chip liquid cooling (Chapter 5.4) gives you a closed loop of clean, conditioned coolant running through cold plates a millimetre above a 1.2 kW die. The building gives you facility water: chilled or warm, conditioned for its own equipment — an open-tower circuit can carry treatment chemicals and dissolved solids the cold plates would not tolerate, while a closed circuit has its own pressure and chemistry limits. These two fluids must exchange heat without ever mixing. The component that enforces that separation — and pumps, filters, monitors, and controls the clean side of it — is the Coolant Distribution Unit (CDU). It is the most operationally consequential box in a liquid-cooled hall that nobody outside the mechanical team thinks about, right up until a pump trips and a training run loses a thousand GPUs to a thermal throttle.

The CDU defines two loops. The primary or facility water system (FWS) is the building side — the loop that runs out to the chillers, dry coolers, or towers (Chapter 5.8). The secondary or technology cooling system (TCS) is the clean side — the loop that runs from the CDU heat exchanger out to the in-rack manifolds and cold plates (Chapter 5.4). The CDU is the membrane between them: a brazed-plate or shell-and-tube heat exchanger, a redundant pump set, a filtration package, an expansion/make-up provision, and a controls stack that holds the TCS supply temperature, flow, and pressure to the accelerator vendor's envelope. ASHRAE TC 9.9 and OCP both codify this FWS/TCS split because it is the line that makes everything downstream tractable: the facility side can be warmer, more variable and less clean than the cold-plate circuit within its own equipment limits; the technology side stays tightly controlled because the connected GPU hardware and its warranty require it.

The membrane raises four decisions. Where you place it (the four CDU architectures), how big and how redundant you make it, what fluid and filtration you run on the clean side, and how you control the dew-point margin and detect leaks. Each carries a downstream cost measured in stranded megawatts, blast radius, or throttled GPUs.

Strip away the marketing and a CDU performs five functions, and you size and select it against all five — not just the headline kW.

• Isolate. The heat exchanger keeps facility water (FWS) and technology coolant (TCS) physically separate. With the plates intact, the two fluid inventories remain separated, although a leak or pressure loss can still interrupt heat transfer. What defeats that is a perforated or fatigue-cracked plate: a cross-leak connects the two circuits in whichever direction the differential pressure pushes, which is why conductivity, level and dP trending is the cross-leak alarm and not a nicety — and why what the facility side actually carries, a closed treated circuit or open tower water, decides what a breach costs you. That isolation is the box's reason for existing.
• Pump. The CDU drives the secondary loop — it provides the flow and head to push coolant through the in-rack manifolds, the ~150-200 quick-disconnects, and the cold plates against their pressure-drop budget. For the HPE GB200 NVL72, a guide water-property heat balance gives ~165–236 L/min through the secondary loop at a 7–10 °C design rise across its 115 kW liquid load; Clariant PG25 gives ~172–246 L/min at the same rise, roughly 4% more (CDU sized ~250–300 L/min). A row or central CDU multiplies that across many racks.
• Control temperature. The CDU modulates a facility-water control valve and pump speed (VFDs) to hold the TCS supply temperature to setpoint inside the GPU vendor's acceptance envelope — for GB200-class racks the published acceptance envelope reaches ~45 °C supply / ~65 °C return (QCT QoolRack acceptance maxima, ASHRAE W45-class — not an HPE figure; Chapter 5.7), with operators picking the setpoint (often ~25-35 °C) to trade free-cooling hours against thermal headroom. The approach temperature of the heat exchanger (how close the TCS supply can get to the FWS supply) is a first-order selection criterion: Google's 2 MW Project Deschutes CDU advertises a 3 °C approach, which directly buys warmer facility water and more free-cooling hours.
• Filter and condition. Full-flow filtration protects the connected equipment at its required cleanliness; a separately specified side-stream stage can polish the inventory; the CDU also hosts make-up, de-gassing, and (often) chemical dosing for the TCS.
• Monitor and protect. Flow, supply/return temperature, differential pressure, conductivity, and leak sensors feed the BMS/DCIM and, increasingly, the cluster scheduler. The CDU is the sensor platform that tells the cluster whether its coolant is in spec before a chip throttles.

There are four common CDU placements; L2L separates FWS from TCS, while L2A transfers TCS heat to air, and the choice is the defining decision of this chapter. They differ on whether facility water reaches the rack at all, on how many racks share one CDU (the blast radius), on serviceability, and on how gracefully the architecture absorbs the density ramp from 132 kW to 600 kW racks.

In-rack liquid-to-air (L2A). A self-contained CDU sits inside or atop the rack and rejects the captured heat back into the room air through an air-cooled heat exchanger. There is no facility water at the rack — the only utility is power and room CRAH/CRAC capacity. This is what makes it the brownfield option: you can land a liquid-cooled rack in a hall that has no building-water connection at the rack; its local liquid circuit still needs containment and commissioning. The cost is capacity and efficiency — an L2A unit is bounded by what the room air can absorb (commonly ~22-100 kW depending on the unit), it dumps the heat back into a hall that now needs more air cooling, and at a ~15 °C approach it is the least thermally efficient option. It is a bridge, not a destination (and it overlaps with AALC in Chapter 5.3).

In-rack liquid-to-liquid (L2L). A CDU in the rack exchanges heat to facility water. Facility water now reaches the rack, but the blast radius is one rack — a CDU failure throttles only its own GPUs. Capacity is bounded by what fits in a few U (often ~50-110 kW per unit, sometimes more), which made it natural for early DLC but is increasingly tight as an HPE GB200 NVL72 alone carries a 115 kW liquid duty. You also pay for many small CDUs and many facility-water drops.

Row-level (in-row) L2L. One larger CDU serves a row of racks. This is the 2026 mainstream for dense AI rows: an MW-class cabinet (Google's Project Deschutes is a 2 MW unit; commercial in-row units run ~600 kW to >1 MW) feeds 10-20+ racks through a row manifold. You amortize one high-quality, internally-redundant CDU across the row, with a CDU-to-rack run kept short (commonly ≤20 m). The fork's cost is blast radius and concurrent maintainability: lose the row CDU without N+1 and you lose the whole row, so the redundancy posture (below) becomes non-negotiable.

Central / plant-level L2L. A few very large CDUs (or a CDU plant) serve a whole hall or pod from a mechanical room, distributing TCS through a building-scale secondary loop. This maximizes amortization, redundancy pooling, and serviceability (the CDUs are out of the white space), at the cost of the largest blast radius, the longest secondary runs (more pumping head, more fluid volume, harder hydraulic balancing), and the heaviest commissioning. It blurs into the facility-loop design of Chapter 5.7.

The L2L-vs-L2A trade comes down to capacity at scale: Vertiv rates its CoolChip CDU 2300 L2L at up to 2.3 MW and its CoolChip CDU 350 L2A at 350 kW at 15 °C ATD — a 6.6× nameplate ratio between two different application ratings, with L2A limited by the room air it rejects into; the quotient does not establish a 6.6× advantage at matched temperature, flow, head or energy boundary. L2A wins only where its singular advantage — no facility water at the rack — is worth more than capacity and PUE, which is exactly the brownfield retrofit case (Chapter 5.10). For a purpose-built training hall whose room cannot accept the L2A duty but whose water path can, the decision narrows to row-level vs central L2L, and that is a blast-radius-vs-amortization argument, not a thermodynamics one.

Sizing a CDU is a four-variable problem, not a matter of picking the kW that matches the rack. The four are heat load, flow, approach temperature, and pressure/head, and you must satisfy all of them simultaneously at the worst-case facility-water temperature, not the design-day average. A CDU rated '2 MW at 18 °C facility water' may deliver far less on the hottest day when the tower can only supply warmer water; the rating is a curve, not a number, and you size against the corner of that curve that your climate and heat-rejection plant actually produce (Chapter 5.8).

The flow side is set by the accelerator: the approved fluid, liquid duty and selected temperature rise, constrained by the OEM minimum-flow curve. The head side is set by the worst-case hydraulic path — the longest, most restrictive run from CDU through row manifold, in-rack manifold, UQDs, and cold plates — and getting it wrong starves the far racks (the equipment-specific branch-flow requirement in Chapter 5.7 is a CDU-plus-manifold co-design problem, not a manifold-only one). The approach temperature is the efficiency lever: a tighter HX approach (3 °C vs 5 °C) lets you run warmer facility water for the same chip inlet, which is what buys free cooling and heat reuse (Chapter 5.9).

The expensive call is the oversize factor against the density ramp. A CDU plant sized exactly for HPE GB200 NVL72 rows at their 115 kW liquid duty has no reserved capacity for VR200 NVL72's 330 kW facility design basis, let alone the ~600 kW Rubin Ultra / Kyber planning point. Buy the headroom now and you carry idle capex and worse part-load efficiency for two years; buy it later and you re-plumb a live hall mid-life. There is no clean answer — the practitioner question 'can you economically future-proof from 120 kW to 600 kW loops, or is a 2-3 year re-fit cycle now structural?' is genuinely open. The defensible move is to oversize the irreversible substrate (pipe risers, valve stations, mechanical-room footprint, facility-water capacity) while keeping the reversible CDU modules matched to current generation — the same reversible-vs-irreversible discipline as the slab and power chain in Chapter 1.1.

The February 27, 2026 Addendum b to ASHRAE 127-2020 supplies a CDU test/rating method. Specify the application temperatures, fluid, flow, available external head and electrical boundary together; state how internal heat and losses enter the selection. A method-compliant rating is not a project-wide acceptance result.

Total exchanger duty = 320 + 3.00 = 323 kW. With heat-capacity rate 32.0 kW/K, the CDU hot inlet is 35.0 + 323/32.0 = 45.1 °C and FWS return is 30.0 + 323/32.0 = 40.1 °C. At the exact unrounded ports both terminal approaches remain 5.0 K. Each 330 kW thermal selection exceeds duty by 7 kW. The auxiliary heat is counted once; neither the 320 kW IT load nor the 330 kW selection is the CDU electrical draw.

A fails hydraulic duty: 200 < 213 kPa. B has approximately 27 kPa head margin and its VFD is trimmed to the required operating point. Select B for the current piping arrangement, carrying the full 323 kW rejection requirement to 5.8. Do not subtract the internal CDU loss from an already external-head rating. Two B units establish steady capacity only; their transfer, power and shared headers still need 13.5 acceptance.

Flip. A becomes hydraulically eligible at external duty ≤200 kPa, provided its thermal and control conditions still hold. The 80.0 mm common-header alternative in 5.13 screens at 191 kPa and crosses that boundary, trading pipe space and capital against CDU selection. Use the ASHRAE CDU rating method to keep the comparison at identical ports.

Because a CDU sits in the goodput path, its redundancy posture must be justified from the selected maintenance and fault states. Where you put it is a fork with different cost and different blast radius.

Internal pump redundancy (N+1 within one CDU). Most quality CDUs ship with dual or N+1 pumps and often redundant power feeds per pump circuit (Project Deschutes runs fully redundant power feeds for each pump circuit), so a single pump or VFD failure does not stop flow. This protects against the most common failure mode — a pump — but the heat exchanger, the cabinet, and the controls are still single points. Lose the box and you lose the row.

Unit-level redundancy (N+1 CDUs). A spare CDU per row (or per N rows) covers the whole-box failure and enables concurrent maintainability — you can valve out and service a CDU without dropping the row. This is the posture that supports concurrent maintenance on the cooling side when isolation, controls, power and surviving flow preserve the required service — a concrete obligation where the row is one training job; Tier III or IV still requires the complete topology and certification boundary in 12.1. The cost is a redundant MW-class cabinet, its facility-water drops, and the floor space.

UPS-backed pumps for thermal ride-through. This is the subtle one. A DLC loop coasts only on usable thermal inventory that maintained circulation can reach: on loss of facility power the coolant stops moving; the selected rack's throttle, controlled-shutdown, emergency-shutdown and no-response states and timing must come from OEM transient data, controls tests or an engineering calculation. The protection path—including whether CDU pumps or rejection plant need UPS/BESS—must then be sized to that validated response and the power-transition case. Skipping this is the cooling-side equivalent of skipping ride-through on the GPUs — see the transient/ride-through treatment in Chapter 5.8 and Chapter 5.12.

The single most important CDU control setpoint is the secondary-supply temperature relative to the white-space dew point. The coolant runs through manifolds, hoses, cold plates, and quick-disconnects — much of it exposed to room air. If any wetted surface drops below the room's dew point, atmospheric moisture condenses on it: water beading on hoses above live 800 VDC busbars and 1.2 kW GPUs. The rule is absolute: hold the TCS supply temperature above the white-space dew point at all times, which keeps the cooling 100% sensible (no condensation, no latent load) and is one of the structural reasons warm-water cooling is winning — a ~45 °C supply has a wide margin in a room whose measured dew point is well below it, provided startup and control undershoot preserve that gap.

This couples the CDU to the room. The dew point depends on white-space humidity, so the CDU's minimum supply setpoint is a function of the air-side environmental control (and of any humidification policy). Run the room too humid and you raise the dew-point floor, forcing a warmer minimum coolant supply and giving up cold-plate margin; run a chilled-water trim loop too aggressively and you risk dipping below dew point on a transient. The dew-point margin is therefore a negotiated setpoint between the mechanical (CDU) and environmental (CRAH/RH) controls, and it is exactly the kind of cross-loop setpoint interaction that Chapter 5.12 treats as a stability problem.

Leaks and pump trips are the fast failure modes. The slow one is the secondary loop fouling itself over a 10–15 year life—and it is the one most likely to be under-engineered at commissioning because it does not show up for years. There is no universal TCS fluid. PG25 is one project-specific choice, and its concentration label is not a complete chemistry specification. The approved basis must name the qualified product, percentage basis, water quality, inhibitor and biocide or biostat package, wetted-material compatibility, operating-temperature range, sampling limits, replenishment or replacement plan, and OEM warranty requirements. The physical trade remains: relative to water at the declared temperature, adding glycol generally raises viscosity and lowers specific heat, increasing pumping power or required flow. Select the concentration and controls from the qualified fluid/equipment/maintenance program, not from a universal default.

Three chemistry failure modes stalk the loop over its life, and the CDU is where you manage all three:

• Galvanic corrosion. Mixed metallurgy (copper cold plates, aluminium components, steel pipe, brazed-plate HX) plus a conductive fluid sets up galvanic cells. Corrosion inhibitors are dosed into the fluid, but inhibitors deplete — so this is a maintenance cadence, not a one-time fill.
• Biofilm. Warm water is a microbial habitat; biofilm fouls cold-plate microchannels and degrades heat transfer. The warm-water roadmap raises this risk exactly as it lowers the chiller bill, so biocide/biostat management gets more important as inlet temperatures climb.
• Particulate / additive depletion. Particulate from manufacturing residue, wear, and corrosion clogs the tightest channels in the cold plates; full-flow filtration (commonly ~50-micron) catches bulk particulate, while a separate side-stream polishing stage can be sub-micron on premium units. Both filter stages load and must be serviced, and additives are adjusted only through the supplier-approved program in 5.7.

The open question the industry has not answered with public data is how often the TCS fluid must actually be replaced over a facility life, and what the real long-run material-compatibility failure rates are — vendor claims need independent validation. The practical consequence: the CDU's filtration spec, fluid-quality sampling cadence, and inhibitor-replenishment regime are commissioning decisions that determine whether you are servicing clogged cold plates in year three. Get the fluid program wrong and the failure shows up as a slow, fleet-wide rise in coolant delta-T and a quiet loss of goodput — the hardest kind to diagnose. Chapter 5.7 defines fluid acceptance; 13.5 accepts the installed cooling system.

A modern CDU is a controls node, and how it integrates decides whether the cooling can defend the compute or merely report on it after the fact. At minimum it modulates pump speed (VFDs) and the facility-water control valve to hold TCS supply temperature, flow, and differential pressure to setpoint, while streaming supply/return temperatures, flow, dP, conductivity, filter state, coolant level, and leak status to the BMS/DCIM. The VFDs themselves matter beyond control: ultra-low-harmonic drives (Project Deschutes specifies IEEE 519 ULHD VFDs) keep the CDU from polluting facility power quality — a real concern on a hall carrying dozens of CDU pump and fan drives alongside the rest of the mechanical plant — a CDU's MW rating is thermal duty, not its drive's electrical input (Chapter 4.12 on power quality).

The decision that separates a goodput-aware facility from a facilities-aware one is whether the CDU telemetry reaches the cluster scheduler, not just the BMS. If the scheduler can see a row's coolant-inlet temperature trending toward the throttle threshold, it can drain or de-rate the job before the GPUs throttle and corrupt a training step; if the data dies in the BMS, the first the cluster knows of a cooling problem is a thermal throttle on a thousand chips. This is the cooling-side instantiation of the DCIM/observability argument in Chapter 14.2 and the autonomy ladder in Chapter 14.13: the CDU's value is not just holding setpoint, it is being a sensor the cluster can act on.

The CDU is the hinge of the whole thermal stack. Upstream of it is the rack-side world — cold plates, in-rack manifolds, and quick-disconnects (Chapter 5.4) — whose pressure-drop and flow-balance budgets the CDU must satisfy. Downstream is the facility water loop and its temperature class (Chapter 5.7), then heat rejection (Chapter 5.8) or heat reuse (Chapter 5.9). The CDU's approach temperature is the dial that connects them: a tighter approach lets the facility loop run warmer, which is what makes year-round free cooling and high-grade heat capture possible. Choose the CDU architecture and approach badly and you have quietly capped your PUE, your free-cooling hours, and your heat-reuse grade before the first GPU boots.

One dial is external to the design: manufacturing capacity. Liquid-cooling plant is now reserved like switchgear — nVent leased its third liquid-cooling factory expansion in three years (160,000 sq ft in Blaine, MN; production 1H 2027; the company reports >2 GW of liquid cooling already deployed, Jul 2026). A CDU selection that ignores the vendor's build slots is a schedule decision made by accident.

Choose the CDU whose thermal duty, external head, controls and isolation envelope all close at the required operating points. A nominally larger exchanger cannot make up missing pump head. Preserve the failure-state duty with the selected topology, then hand the installed response to Chapter 13.5.