Facility Water Loops & Warm-Water Cooling

The CDU isolates the technology-cooling loop at the rack from the building. Everything on the other side of that heat exchanger — the pipes in the pipe rack, the pumps in the plant room, the water that ultimately carries the GPUs' heat to the sky — is the facility water system (FWS), and it is governed by one decision that sits upstream of almost every mechanical choice in the building: how warm is the water you push to the CDUs? That single setpoint is not a tuning parameter you adjust in operations. It is a design-basis commitment that selects your heat-rejection plant, sets how many hours a year a compressor has to run, fixes your water-versus-energy position, and decides whether the heat coming back is a sellable commodity or a thermodynamic nuisance you pay to dump.

The loop temperature is the master fork of this chapter. We map the ASHRAE liquid classes (W17 through W45+) and what each commits you to; trace why warm-water cooling — once exotic, now the 2026 default for GB200/GB300-class racks — unlocks year-round free cooling and high-grade heat reuse; and then work the engineering the temperature choice forces downstream: loop topology and flow balancing, the delta-T discipline the whole plant lives or dies on, water chemistry and treatment over a 10–15 year life, and the fill/flush/commissioning sequence. Heat rejection itself — the chillers, towers, dry coolers, and economizers the loop couples to — is Chapter 5.8; the economics of selling the heat are Chapter 15.5. Here we engineer the water that connects them.

ASHRAE TC 9.9's liquid-cooling guidance bands the facility water loop the way its air classes (A1–A4) band the supply air: each class is named for the maximum allowable supply (entering) water temperature the equipment is rated to accept. The 5th-edition refresh renamed and extended the ladder to keep pace with AI density, adding the warm classes the industry is now standardizing on. The numbers are the upper supply limit in °C; every class shares a 2 °C lower bound to keep the loop above any plausible dew point at the CDU.

Each class implicitly specifies a heat-rejection strategy. W17/W27 are the legacy chilled regime: you need a chiller and a tower because the supply has to be colder than ambient. W32/W40 are the inflection: in most climates you can run chiller-less for the great majority of the year. W45/W+ are the warm-water target the current accelerator generation was designed around — chiller-less by intent, with the heat hot enough to reuse. Reading the class off a datasheet tells you, before any plant sizing, roughly what the mechanical room and the water bill are going to look like.

The strategic content of the table is the diagonal: as you walk warmer, the compressor leaves the critical path and the waste heat becomes worth something — but the thermal margin between your supply water and the chip's junction limit shrinks. A W45 loop feeding a GB200 NVL72 (rated to accept liquid up to ~45 °C inlet, returning up to ~65 °C) is operating with almost no slack: there is no colder water to fall back on if a cold plate fouls or a CDU approach drifts. That is the warm-water bargain stated plainly — you trade away your thermal insurance to delete the chiller. → the junction-to-coolant resistance budget that makes this tight is Chapter 5.4; the CDU approach that consumes part of the margin is Chapter 5.6.

Warm-water cooling is the rare decision that improves two normally-opposed objectives at once, which is why it went from contrarian to default in roughly three years. The mechanism is simple thermodynamics: heat rejection is free whenever the loop is warmer than the ambient you are rejecting into. A 18 °C chilled loop is colder than summer air almost everywhere, so a compressor must lift the heat uphill — that is the energy, the capex, and the single largest mechanical failure mode in the building. Raise the loop to 45 °C and it is hotter than the design-day air across nearly the entire populated world, so a dry cooler or tower rejects directly and the compressor is a backstop that may never run. The cooling plant's share of facility energy — commonly 30–40% in air halls — collapses toward the parasitic load of pumps and fans.

The second prize is the heat itself. A chilled loop returns water at ~25–30 °C — lukewarm, useful for almost nothing, requiring an enormous heat-pump lift to reach the 60–80 °C a district-heating network wants. A warm W45 loop returns at ~55–65 °C, which is close enough to district supply that the heat-pump lift is short, the coefficient of performance is high, and the waste heat crosses from cost to commodity. This is the engineering reason the EU's heat-reuse mandates are tractable at all: Germany's Energy Efficiency Act (EnEfG) requires data centers commissioned from 1 July 2026 to hit an Energy Reuse Factor of at least 10%, rising to 15% in 2027 and 20% in 2028 — a quota that is feasible with warm-water DLC and effectively impossible with a chilled-air hall. → the offtake economics and the chicken-and-egg of finding a heat buyer are Chapter 15.5; the engineering of the lift is Chapter 5.9.

The facility loop's job is to move a fixed quantity of heat from a few hundred CDUs to the heat-rejection plant while holding every CDU's supply temperature within a tight band, regardless of how the IT load shifts across the hall. Two topology forks define how you do it, and each has a characteristic failure mode.

Primary/secondary (decoupled) vs variable-primary. The classic mission-critical pattern decouples a constant-flow production loop (pumps through the chillers/heat exchangers) from a variable-flow distribution loop (pumps to the CDUs) via a low-loss header or decoupler, so the rejection plant sees stable flow while the distribution side modulates to load. It is robust and easy to stage, but it carries a second set of pumps and their parasitic energy. Variable-primary-flow collapses the two into one variable loop with bypass control — fewer pumps, lower parasitic load, lower capex — at the cost of a more demanding control problem: the plant must protect minimum flow through the rejection equipment as the distribution side throttles. For a warm-water DLC plant whose whole premise is energy efficiency, the parasitic-pump saving of variable-primary is attractive; for a plant that must ride through synchronized GPU load slams without the thermal inertia of a chilled-water buffer, the predictability of decoupled flow is worth the pumps. → the transient/ride-through dimension of this choice is Chapter 5.12.

Flow balancing across the hall. A loop feeding 200 CDUs is a hydraulic network in which the nearest CDU wants to hog flow and the farthest is starved. Reverse-return piping (Tichelmann) equalizes path length so every branch sees similar pressure drop; direct-return is cheaper but needs balancing valves and commissioning effort to avoid the far racks running hot. The branch-balance target at the rack manifold is tight — on the order of ±5% — because an out-of-balance branch shows up as a hot GPU, not just an inefficiency. → the in-rack manifold side of this is Chapter 6.1; the worst-case-branch flow validation at commissioning is Chapter 13.5.

Every pipe diameter, pump size, and heat-exchanger area in the facility loop is sized against one assumed number: the design temperature rise (delta-T) between supply and return. The relationship is fixed by physics — for a given heat load, flow is inversely proportional to delta-T. Design for a wide delta-T (say 12 °C) and you move the heat with less water, so pumps and pipes are smaller and cheaper; design for a narrow delta-T and everything grows. Warm-water DLC plants target roughly 7.5–12 °C across the loop, with ~1.2–2.0 L/min per kW of PG25 coolant on the secondary side.

The fork is not the design number but what happens when you fail to achieve it. Low-delta-T syndrome is the chronic disease of large chilled/warm-water plants: through three-way-valve bleed-by, oversized coils, dirty heat exchangers, or simple control sloppiness, the loop returns colder than designed, so the plant pushes more flow to move the same heat, so the pumps run harder and the rejection equipment loses efficiency — and the symptom compounds because higher flow further shrinks the delta-T. The consequence is a plant that hits its flow limit long before its thermal limit and strands cooling capacity you paid for. The discipline is to design for an honestly-achievable delta-T, use two-way valves and proper sequencing, and meter return temperature per branch so the syndrome is caught as drift rather than discovered as a capacity wall. → the metering that makes this observable is Chapter 4.12; the controls tuning that prevents the valves from hunting is Chapter 5.12.

The facility loop is a closed mixed-metal system — steel pipe, copper and brass fittings, aluminum and copper cold-plate alloys downstream of the CDU heat exchanger, stainless plates — circulating treated water or a water/glycol mix for the life of the building. Left untreated it does three things, all of which eventually end at a fouled cold plate and a throttled GPU: it corrodes, it grows biology, and (if glycol is present) it oxidizes into acid. The treatment regime is not optional housekeeping; it is the thing standing between a clean loop and a slow-motion capacity loss.

Corrosion. A mixed-metal loop sets up galvanic couples; the defense is an inhibitor package (nitrite for ferrous-dominant systems at ~600–1,200 ppm, molybdate or phosphate blends for mixed-metal circuits) holding pH in roughly the 8.5–10.5 band. Dissolved iron above ~2 ppm is the early signal of active corrosion — corrosion products are also the particulate that fouls the ~50-micron CDU filters and the cold-plate microchannels. Biology. Warm water is a thermophile incubator; a closed loop still needs biocide and biofilm control, because biofilm both insulates heat-transfer surfaces and harbors microbiologically-influenced corrosion. (The Legionella exposure lives on the open-tower side of the plant, governed by ASHRAE 188 — that is a Chapter 5.8 problem.) Glycol oxidation. Where freeze protection or biostasis mandates propylene glycol (commonly PG25), the glycol slowly oxidizes in the presence of oxygen, heat, and metal catalysts into glycolic, lactic, and formic acids; once pH falls below ~7 the inhibitors are spent and the loop turns corrosive. The 10–15 year question — how often must the TCS/FWS fluid actually be replaced — is still being answered in the field, which is exactly why monitoring matters more than any single additive.

The 2026 best practice is continuous real-time sensing of pH, conductivity, and inhibitor reserve, with monthly lab confirmation of dissolved metals — closing the data gap that quarterly grab-samples leave open, so chemistry swings are caught in minutes rather than months. The consequence of skipping it is not abstract: a corroded, fouled loop strands cooling capacity inside a building whose entire economic case is keeping GPUs at full clock.

The facility loop is a charged pressure system that must be clean at fill, leak-tight at pressure, and balanced at flow before a single GPU is energized — and the order of operations is not negotiable. Piping design follows from the delta-T and flow decisions above: pipe is sized to a velocity that moves the design flow without erosion-corrosion (typically a few m/s ceiling) or excessive pressure drop, materials are selected and dissimilar metals isolated to prevent galvanic attack, and the routing reserves pipe-rack space and knockouts for the density ramp the building will see. The structural, surge/water-hammer, NPSH, and code-basis (ASME B31.x / EN 13480 / PED) mechanics of the charged pipe are deep enough to be their own chapter — Chapter 5.13 — and the weight of charged pipe on the slab is Chapter 6.2.

Fill and flush is where AI-density loops differ most from legacy chilled water. A loop that feeds sub-millimeter cold-plate channels cannot tolerate the mill scale, flux, weld slag, and cutting debris that a new piping system always contains; left in, that debris fouls the CDU filters and the microchannels within weeks. The protocol is a staged chemical clean and a multi-pass flush — commonly a high-velocity flush to dislodge debris, a chemical cleaning/passivation pass, and repeated deionized-water flushes (factory-integrated racks see ~3x DI flushes over 12–16 hours before fill) — verified by fluid quality (turbidity, conductivity, particle count) rather than by calendar. Only then is the loop charged with treated fluid, vented of air (entrained air kills pump NPSH and heat transfer), and pressure-tested. Commissioning then proves the loop end-to-end: leak-tight at test pressure, every branch flowing within tolerance under worst-case-branch conditions, delta-T achieved at load, and leak-detection response time within acceptance. → the formal L1–L5 cooling-acceptance ladder, the flush acceptance criteria, and the worst-case-branch flow test are Chapter 13.5; the leak-detection and serviceability engineering is Chapter 5.11.

The recurring facility-loop mistakes all come from treating one of its design-basis numbers as adjustable later:

• Chilled-by-default for margin you didn't price. Specifying a W17/W27 loop because the chip datasheet allows it and it feels safe — and silently committing the building to a compressor in the critical path, an evaporative tower's water bill, and waste heat too cold to sell. If the silicon accepts W45, going chilled is buying thermal insurance at the price of the plant's whole efficiency case.
• Designing a wide delta-T you can't hold. Sizing pumps and pipes for a 12 °C delta-T and then operating at 6 °C because of valve bleed-by and dirty coils — low-delta-T syndrome that doubles the flow, strands cooling capacity, and surfaces as a hot-rack wall long after the plant is built.
• Skimping the flush. Treating the fill/flush as a legacy chilled-water plumbing step rather than a sub-millimeter-channel cleanliness gate — and seeding the loop with weld slag and mill scale that fouls the CDU filters and cold plates within weeks of go-live, converting a commissioning shortcut into a chronic goodput tax.
• Treating warm water as free ride-through. Banking the chiller-less efficiency while forgetting that you deleted the chilled-water thermal inertia that used to ride through an upset — and discovering at the first CDU trip that seconds of cooling loss is now a hardware-damage event.