Chapter 1.7
In this chapter · 6 sections
The Requirements-and-Consequences Matrix
Name the workload profile, then turn its measured demands into a signed design basis. Cooling, fabric, storage, redundancy and siting each need their own evidence about the equipment, facility, traffic and service level. The workload label alone selects none of them.
What you'll decide here
- Air, rear-door or direct-to-chip liquid. Heat split and inlet limits pick it before steel is cut, and a retrofit cannot cheaply undo it.
- Cooling limits for the rack, the back-end blocking ratio, and the GPU:CPU, memory and storage ratios, reset from measured heat, traffic and service targets.
- The storage tier and its throughput floor (checkpoint write bandwidth, data-loader read bandwidth, KV-cache capacity), set by how much stalled GPU time the archetype tolerates.
- Whether a restart-tolerant single path is enough, or the load must survive maintenance and one fault. The recovery limit decides.
- Whether the site is scored power-first or latency-first, and the reference design-basis sheet per archetype that you freeze and sign before ordering long-lead equipment.
Chapter 1.1 set the governing input, the workload archetype, and walked the cascade in qualitative terms. This chapter turns it into an engineering instrument: a requirements-and-consequences matrix that takes the measured workload, the service limits and the equipment you have chosen, and records a numbered design basis for every subsystem that follows. Where 1.1 set the workload fork, this chapter asks for the evidence behind each cell: inlet temperature, flow rate, floor-loading class, blocking ratio, storage throughput floor and redundancy tier, plus what each cell costs downstream if you fill it in wrong.
The chapter moves through four mappings in the order an engineer commits them: density to the cooling cliff, fabric and the GPU:CPU/memory/storage ratios, storage and redundancy against interruption tolerance, and siting as power-first versus latency-first. It ends with the reference design-basis sheets that capture all four per archetype. Keep Chapter 1.1 open beside it: this is the table its cascade was promising.
Mapping 1 — rack density and heat split to the supported cooling envelope
The first irreversible commitment is the cooling modality, and rack density narrows it. Rear-door heat exchangers (RDHx) carry a typical 30–40 kW rack, and past 50 kW with active doors. Above that, the rack's heat split decides whether direct-to-chip liquid is the only path. The bands overlap, so heat flux, airflow and inlet limits, water interfaces and residual room heat make the call. RDHx can extend an existing air architecture by bringing chilled or tempered water to the door. An AALC or in-rack L2A sidecar keeps the liquid loop local and rejects the heat to room air instead, which trades the door's water drop for more room-air heat rejection. Direct-to-chip liquid (DLC) becomes the normal path when the rack's profile and heat flux require liquid capture at the source.
The current racks leave little room for argument. An HPE GB200 NVL72 has a 132 kW nominal rack TDP, roughly 115 kW removed by liquid and 17 kW by residual air, so it is a liquid-plus-residual-air architecture by definition. GB300 NVL72 shipped in 2025 and deploys through 2026 alongside GB200; Lenovo specifies 135 kW TDP, up to 155 kW peak, and a ~90/10 liquid/air split. VR200 NVL72 has been shipping since June 2026. NVIDIA's DSX reference design plans its cabinets at 330 kW TDP, the facility basis for power and cooling, while Pegatron rates its operating profiles at 188 kW Max Q and 228 kW Max P. Size facility power and cooling reserve to 330 kW if you follow NVIDIA's reference design, and use 188/228 kW for expected operating load. Rubin Ultra / Kyber remains a ~600 kW roadmap planning point on an 800 VDC path. Two racks at the same draw can still need different services, because their heat split, flux, airflow limits and water interfaces differ. The matrix below therefore records the evidence for each rack, not just its kilowatts.
A future liquid-cooled profile can strand capacity where the inherited hall lacks the structure, distribution, isolation, CDU space or heat rejection it needs. Price that conversion against its actual scope; Chapter 5.10 owns the retrofit comparison. The engineering lives in Chapter 5.1 through Chapter 5.4.
| Decision input | Evidence to record | What it can rule in or out | Downstream consequence |
|---|---|---|---|
| Rack thermal profile | Rack kW, component heat flux, liquid/residual heat split, OEM-supported thermal interfaces | Whether source capture is required and how much room-air duty remains | Sets rack, manifold, CDU, airflow, and residual-room loads |
| Supported room-air limits | Required airflow and static pressure, inlet range, containment/recirculation, acoustics, fan state, service access | Whether conventional or close-coupled air can carry your rack | Sets fan power, floor/aisle allocation, controls, and operating margin |
| TCS/FWS and entering-water conditions | Coolant, supply/return limits, design delta-T, flow, pressure drop, CDU approach, isolation and redundancy | Whether RDHx or direct liquid works at the supported operating point | Sets piping, CDU, pumping, controls, and maintenance boundaries |
| RDHx/AALC rating | Model-specific capacity at the rated air and water temperatures, flow, pressure, fan state and capture target; full residual-room rejection for AALC | Whether the bridge works without exceeding room-air or water capacity | Adds door/sidecar fan, water, condensate, acoustic, and failure obligations |
| Climate and heat rejection | Design-day ambient, economizer hours, water constraint, rejection approach, heat-reuse duty | Which supply points and rejection systems remain feasible | Sets plant capex, PUE/WUE, water use, and reuse grade |
| Service, redundancy and refresh tail | Isolation, maintainability, fault response, spare strategy, future rack profiles | Whether today's feasible path remains operable and expandable | Sets irreversible structure, routes, capacity reservations, and stranded-asset risk |
Mapping 2 — fabric sizing and the system ratios
The second mapping is the network, and it has two parts: the blocking ratio of the back-end (scale-out) fabric, set by coupling, and the system composition ratios (GPU:CPU, GPU:memory, GPU:storage), set by the archetype's host-side and data-path demands. The wrong answer wastes money in opposite directions. Over-build and you pay for bandwidth that never carries traffic; under-build and you starve the accelerators you spent the most on.
Coupling sets the blocking ratio. Synchronous training keeps the rail tier non-blocking, because the data-parallel all-reduce crosses it on every step. Inference runs 2:1–3:1 at the upper tier in some published designs, because request and KV traffic is bursty and mostly leaf-local. The tiers above either can run far leaner: SemiAnalysis reports 7:1 at the top of Meta's 24,000-H100 training cluster. SemiAnalysis's 3 October 2024 model for a 1,024-H100 cluster estimated 31.6% lower back-end network cost when 2:1 oversubscription was combined with its virtual modular switch design, against 24.9% for the non-blocking virtual modular design alone. Chapter 8.5 prices the actual port and equipment list and gives the three tests that say whether a tier may take that saving: locality, exposed communication and failure headroom. Wide expert parallelism or prefill/decode disaggregation spilling across racks pushes a tier back toward 1:1. → Chapter 8.5 (topology & oversubscription), Chapter 8.4 (protocols).
The composition ratios depend on the archetype and are shifting. A GPU:CPU ratio must name its package denominator and its hardware. A DGX B300 has eight GPUs and two CPU packages (4:1), while GB200/GB300 and Vera Rubin NVL72 racks have 72 GPUs and 36 CPU packages (2:1), for training and serving alike. TrendForce's 14 April 2026 forecast moves deployment-level CPU:GPU ratios from 1:4–1:8 toward 1:1–1:2 as agentic work (sandbox execution, retrieval, tool calls, RL rollouts) adds CPU demand, and Intel describes the same direction. That changes the host BOM and the node power budget. Size attached hosts and any separate CPU pool from their measured work; Chapter 7.8 owns that composition decision.
GPU:memory is set by per-GPU HBM (H100 80 GB, B200 180 GB, B300 288 GB, Rubin Ultra ~1 TB at the package level, roadmap) plus host RAM, and inference is increasingly KV-cache-bound rather than weight-bound. GPU:storage sizing records both capacity and service. The dataset and model footprint, checkpoint retention and KV/staging allocations must fit, while loader reads, checkpoint capture and drain, restore and KV transfers must meet their time budgets. Chapter 9.1, Chapter 9.4 and Chapter 9.7 own those methods; Mapping 3 records the resulting floors.
| Archetype | Back-end blocking ratio | GPU:CPU packages (by hardware) | Dominant memory pressure | Storage demand profile |
|---|---|---|---|---|
| Pre-training | Typically 1:1 non-blocking, 8-rail fat-tree; confirm against measured collective traffic | DGX B300 4:1; NVL72 2:1 | HBM for activations; host RAM for staging | Burst checkpoint writes; high sustained read for data loader |
| Post-training / RL | Disaggregated: tight trainer, tolerant rollout pool | Mixed — more CPU on rollout side | KV-cache on rollouts; HBM on trainer | Rollout reads + trainer checkpoints; staleness-tolerant |
| Online inference | Often 2:1–3:1 oversubscribed, sized to the tail-latency SLO and measured traffic matrix | DGX B300 4:1; NVL72 2:1; size agentic CPU demand on attached hosts or a separate pool from measured work | KV-cache capacity & bandwidth | Model-weight load; KV-cache tiering to NVMe/CXL |
| Batch inference | Heavily oversubscribed; cost-optimized | Flexible | Throughput over latency; large batches | Throughput reads; no low-latency requirement |
| Edge inference | Minimal (single node / WAN backhaul) | Constrained by appliance | Single-model resident; small KV | Local model store; periodic sync |
Scope & caveats
Select on the named door/rack, air and water conditions, fan state, containment, heat-capture target, residual room heat, climate/rejection, serviceability, redundancy, and future density.
Reference capacity, not a universal ceiling; verify named door/rack, water and air conditions, fan state, containment, and capture target.
Scope & caveats
Exact HPE product profile. Keep this 132 kW / 115 kW liquid / 17 kW air record separate from the OCP MGX Rev. 1.1 reference profile of 120 kW / approximately 102 kW liquid / 18 kW air.
Scope & caveats
Exact QCT reference. The 45 °C liquid-inlet maximum and 65 °C liquid-return maximum are separate limits, not a prescribed 20 K operating rise. Select a supported operating point, approved fluid, liquid heat load, and design ΔT; ASHRAE W45 describes FWS supply capability, not this product's setpoint.
Separate acceptance maxima, not a prescribed 20 K operating rise; do not attribute these limits to HPE without an HPE document that states them.
Scope & caveats
named reference designs and deployments with different traffic, topology, placement and service objectives
Derive oversubscription from the measured traffic matrix, collective/request mix, topology, failure headroom and SLO; validate it on the target fabric.
Scope & caveats
Analyst forecast for deployments; counting methodology is not specified and it is not an in-node CPU-package requirement
Scope & caveats
Enabled Energy, July 2025, compares this scope with $8–10M/MW greenfield in constrained markets; it does not specify liquid-ready operation or price a 100 kW+ conversion
Scope & caveats
Secondary-source whole-facility planning estimate quoted by Savills in May 2024; no disclosed estimating population or method. No universal multiplier follows from Uptime Tier criteria.
Single-source planning heuristic: Savills (May 2024) attributes it to Dgtl Infra, which publishes no sample, geography, density or estimating method. The older ~10–25% inverts a McKinsey 2011 statement (10–20% saving moving Tier IV→III). Uptime Tiers are outcome-based, so no universal cost multiplier follows from the standard; re-estimate against the actual design.
Scope & caveats
Stipulated endpoints for sensitivity only. They are neither measured provider outcomes nor universal targets. Chapter 14.1 reconciles productive-time boundaries; a site measures its own baseline.
Mapping 3 — storage and redundancy against interruption tolerance
Storage and redundancy are two consequences of the same input, the archetype's tolerance for an interrupted GPU, and they are most defensible when designed together. Storage answers one question: when does a GPU stall waiting on data, and what does the stall cost? Redundancy answers another: when a node or a power feed fails, does the workload restart cheaply or lose money?
Storage is sized by the throughput that keeps GPUs fed, not by capacity alone. For training, the two binding flows are checkpoint service (capture, durable write and restore) and data-loader read bandwidth. Synchronous saves block through durable completion, while asynchronous saves resume training after capture and drain in the background, so size both the measured pause and the protected-state age (Chapter 9.4). A starved loader idles the whole pipeline. A high-bandwidth parallel file system feeding GPUDirect Storage (CPU-bypass) is the training default, and it is the link that turns a storage decision into a GPU-efficiency decision (Chapter 9.1, Chapter 9.3, Chapter 9.4). For online inference, the new pressure is the KV-cache: reasoning models emit long decode sequences, which inflate per-request cache, so the hierarchy now tiers KV state across HBM, host memory and NVMe/CXL (Chapter 9.7). Batch and edge are the relaxed cases, with throughput reads and no low-latency floor.
Redundancy is set by interruption tolerance, and over-building it is a recognizable waste. A synchronous training job already restarts from a checkpoint when any node fails. SemiAnalysis reported ~7 days of MTBF for one 512-H100 cluster at a top-tier operator, while Meta's Llama 3 405B run on 16,384 H100s logged roughly one interruption every three hours. The two observations are defined differently and are not one scaling curve. A single-path or component-redundant posture is therefore viable when checkpoint recovery meets the recovery requirement (Chapter 1.1's over-provisioned-redundancy anti-pattern: a stronger topology must justify its cost through a required outcome). An always-on inference business often inverts this. When maintenance or one component fault would breach the SLA and lose revenue, require no load loss for either event. → Chapter 12.1 (redundancy topologies), Chapter 12.2 (goodput vs availability), Chapter 12.4 (goodput SLAs).
| Archetype | Interruption tolerance | Binding storage flow | Storage tier | Redundancy posture |
|---|---|---|---|---|
| Pre-training | High — checkpoint-and-resume | Checkpoint write + loader read bandwidth | Parallel FS + NVMe; GPUDirect Storage | Restart-tolerant; a single path is acceptable when checkpoint recovery meets the objective |
| Post-training / RL | High — staleness-tolerant, restartable | Rollout reads + trainer checkpoints | Tiered: fast trainer FS + rollout object store | Restart-tolerant; separate trainer and rollout fault domains, then size redundancy to recovery limits |
| Online inference | Low — outage = lost revenue + SLA breach | Weight load + KV-cache bandwidth | KV tiered HBM→host→NVMe/CXL | No load loss for maintenance or one component fault when either event breaches the serving SLO |
| Batch inference | High — queue-and-retry | Throughput reads | Object store / capacity tier | Queue-and-retry tolerant; a single path is acceptable when delay stays inside the objective |
| Edge inference | Site-level — fleet geo-redundancy | Local model store + periodic sync | Local NVMe; minimal | Per-site interruption is acceptable when fleet routing preserves the latency objective |
Deep dive: why checkpoint bandwidth and facility redundancy are the same decision
It is tempting to file checkpoint storage under "storage" and redundancy under "electrical" and size them in separate workstreams. For training, that separation hides the trade. A synchronous job's resilience strategy is checkpoint-and-resume: a recoverable node failure is absorbed by reloading a surviving checkpoint and replaying. The strategy has two costs. One is the goodput lost during blocking capture or a synchronous save, plus interference from any background drain. The other is the work re-done since the last checkpoint. Faster durable writes can reduce a synchronous pause or an asynchronous commit lag, so you can checkpoint more often and re-do less work. Chapter 9.4 tests the save interval, protected-state age, drain backlog and restore time together.
Checkpoint bandwidth and facility redundancy are therefore two prices for the same thing, tolerated interruption, and should come from one budget. For a checkpointable job, compare faster checkpoints with a second power path against the same allowed interruption and completion budget. Chapter 12.2 ranks the two in one project model, and the recovery limit in the contract is what flips the order. → Chapter 9.4, Chapter 12.2.
Mapping 4 — qualify power-first and latency-first sites
The fourth mapping is the least reversible of all, because you cannot move a slab. Derive it from the workload; never choose it first and rationalize it after. Latency sensitivity is the discriminator. Pre-training and batch inference are indifferent to user proximity, so they are scored power-first: chase the cheapest firm (or curtailable) megawatts and the coldest free-cooling climate, accept that the site may be hours from any metro, and treat the grid-interconnection queue slot as the scarcest asset in the project. Online and edge inference are scored latency-first: chase sub-50 ms reach to users and accept dearer power, distributing capacity for proximity rather than concentrating it for cost. Metro power typically costs ~1.5–2x the cheapest US industrial rates (EIA, July 2026), with Virginia at 10.07 ¢/kWh against 7.07 in Texas and 6.08 in Louisiana. Only California- or New Jersey-type markets reach 2.5x or more.
The 2026 context sharpens the fork. The binding constraint is often power, not chips. Large-load service dates depend on the utility, its tariff, studies and upgrades, and on the project itself, so a power-first archetype sited near expensive, constrained metro power burns both money and a queue slot it cannot recover. A latency-first archetype in a cheap-power exurb may meet its energy budget and miss its SLO, which is the more expensive miss because it loses the revenue the building exists to earn. Facility water is a siting gate for water-dependent cooling topologies. Where it is unavailable, choose liquid-to-air CDUs or closed-loop dry heat rejection if the ambient range allows (Chapter 3.7). The reordered hierarchy and the speed-to-power race are engineered in Chapter 3.1 and Chapter 3.2; the fiber/latency screen is in Chapter 3.6.
The reference design-basis sheet, per archetype
The four mappings converge into a single artifact: a reference design-basis sheet for each materially different workload and equipment profile, which freezes the accepted assumptions before any long-lead equipment is ordered. This is the "design-basis document" that 1.1 promised, shown below as an illustrative record. Each sheet pins one row per subsystem: density tier, cooling modality, fabric blocking ratio, system ratios, storage tier and throughput floor, redundancy topology, voltage class and siting class. It also carries a reversible-vs-irreversible register that records which assumptions are hedged and which are committed. The table below is the skeleton. A real sheet attaches the numbers (the ramp curve, the MVA sizing, the CDU capacity) plus the acceptance evidence, owner and approval state.
| Subsystem | Pre-training | Frontier inference (NVL72-class) | Enterprise inference (8-GPU node) | Batch inference | Edge inference |
|---|---|---|---|---|---|
| Density tier | 100–600 kW (DLC) | Lenovo GB300 NVL72: 135 kW TDP / up to 155 kW peak; NVIDIA RA facility basis up to 142 kW; Vera Rubin NVL72: 330 kW DSX design TDP (power and cooling basis); 188 kW Max Q / 228 kW Max P operating (Pegatron) | 14.5 kW consumption / 15 kW max per DGX B300 node | 30–60 kW (flexible) | 5–50 kW networked edge DC class |
| Cooling modality | DLC, warm-water loop | DLC, warm-water loop | Air or DLC per OEM system and site envelope | Host hall's existing (air often fine) | Air / sealed modular |
| Fabric blocking | Typically 1:1 non-blocking (confirm on measured collectives) | Sized to the tail-latency SLO and the measured traffic matrix, not to the workload label | Often 2:1–3:1 (SLO- and traffic-sized) | Heavily oversubscribed | Minimal / WAN backhaul |
| Storage tier | Parallel FS + GPUDirect | KV-tiered HBM→host→NVMe/CXL | KV-tiered HBM→NVMe | Object / capacity tier | Local NVMe |
| Redundancy | Restart-tolerant; single path acceptable if recovery meets the objective | No load loss for maintenance or one fault when the SLA requires it | No load loss when the SLA requires it; otherwise fleet routing absorbs a node loss | Queue-and-retry tolerant; single path acceptable | Per-site interruption acceptable if fleet routing preserves latency |
| Voltage class | 415/480 VAC → 800 VDC path | 415/480 VAC → 800 VDC path | 415/480 VAC | 415/480 VAC | Local LV / appliance |
| Siting class | Power-first (cheap, cold, big queue) | Power-first or metro-anchored campus | Latency-first (sub-50 ms) | Cheapest / curtailable power | Proximity over cost |
A filled design basis for a small serving fleet
Work through the normal and failed states. Eight nodes × eight GPUs = 64 installed GPUs. A node holds 70 × 10⁹ × 2 = 140 GB of weights. Weights + KV + buffers = 140 + 120 + 20 = 280 GB, against 8 × 80 = 640 GB of aggregate HBM. That passes the aggregate screen, though tensor placement, temporary peaks and the 8,000-token cap still need a runtime test. Installed host capacity is 16 CPU sockets, 4,096 GiB RAM and 16 TB local staging. Storage acceptance must demonstrate model loading, restart and request-log durability within the recovery requirement. Local staging capacity is not a durable checkpoint guarantee.
Peak offered attempts = 10 × 1.10 = 11 attempts/s. Healthy candidate throughput is 8 × 2 = 16 attempts/s; with one node unavailable it is 7 × 2 = 14 attempts/s, leaving 3 attempts/s of headroom. At the assumed 1 MB/attempt, peak external traffic is 11 MB/s (about 0.09 Gb/s), below one surviving 100 Gb/s attachment. This screen does not test KV transfers, model distribution or tail behavior. Do not admit a cross-node model partition in this profile without reopening the fabric basis.
IT demand = 8 × 12 + 4 = 100 kW, or 25 kW/rack against the assumed 30 kW rack limit. With one cooling unit out, 2 × 60 = 120 kW remains against 100 kW of IT heat. Coincident facility demand = 100 + 28 = 128 kW, and apparent demand = 128/0.95, about 1.3 × 10² kVA; one surviving 160 kW/200 kVA path passes both screens. An independent path must reach the rack and its controls. The inlet, airflow, heat-rejection state and safe isolation route stay in the acceptance record, and an annual PUE cannot prove any of them.
Select the eight-node candidate for qualification, with procurement on HOLD. The service owner must produce the offered trace and quality evaluation. The platform supplier must prove memory placement, node rate and the supported rack limit. The facility engineer must prove power, cooling, floor load and the delivery route. Operations must demonstrate drain, retry, fault detection and restoration. Release follows their evidence, not the arithmetic alone. The CPU and staging assumptions are ceilings to test, not ratios the workload label guarantees.
Flip: with one node unavailable, the node-rate crossover is 11/7, about 1.6 attempts/s. At 1.5 attempts/s, seven survivors deliver 10.5 attempts/s and miss demand by 0.5. Nine installed nodes leave eight survivors and 12 attempts/s, but the extra 12 kW node also needs a new placement: a third node on one existing rack would push it from 25 to 37 kW, beyond its 30 kW limit. Add a qualified rack, lower the admitted demand or use a faster qualified configuration; do not solve serving capacity by overrunning the cooling basis. MLCommons supplies the primary quality-and-latency qualification pattern. The method homes are Chapter 7.6 for memory, Chapter 5.1 for heat, Chapter 8.5 for fabric and Chapter 13.5 for integrated acceptance; Chapter 1.8 prices the resulting paid fleet.
Deep dive: reading the matrix backwards to audit an existing facility
The matrix is written forward, archetype in and design basis out, but its most useful diagnostic mode is backward. Take a facility that already exists (a hall you are evaluating to lease, retrofit or acquire), read its observable subsystems back up the cascade, infer the archetype it was built for, and compare that with the workload you intend to run.
A hall whose 40 kW rack profile works under recorded airflow and inlet conditions, with an oversubscribed Ethernet fabric, duplicated power and a metro location, may suit a latency-sensitive serving objective. A different training rack may fail that inherited thermal limit, starve the all-reduce and pay for redundancy beyond its recovery requirement. A campus with NVL72-class DLC racks and a non-blocking InfiniBand fabric no longer identifies its own workload, because frontier serving deploys the same racks and disaggregated prefill/decode plus MoE all-to-all consume real bisection. The discriminators sit elsewhere: fabric scale and the blocking ratio per tier, whether storage is sized for checkpoint bursts or for KV-cache tiering, and whether the failure model assumes restart-from-checkpoint or live request retry. Component redundancy and a remote cheap-power site still read as restart-tolerant compute. The same racks in a metro hall that tolerates no load loss for a single fault read the other way. The mismatches a backward read exposes are the three anti-patterns 1.1 named: training fabric for an inference business, forcing a rack beyond the inherited cooling limit, and over-provisioned redundancy for checkpointable jobs. The matrix is both a scoping tool and a due-diligence checklist, the same table run in two directions. → Chapter 5.10 (retrofit limits), Chapter 1.6 (procurement diligence).
Cite this chapter
Fehn, J. (2026). The Requirements-and-Consequences Matrix (Chapter 1.7). The Definitive Guide to AI Data Centers. https://aidatacenterguide.com/part-1-strategy-workload-archetypes-and-economics/1-7-the-requirements-and-consequences-matrix (accessed 2026-10-05).
@misc{aidc-1-7,
author = {Fehn, Jacob},
title = {The Requirements-and-Consequences Matrix (Chapter 1.7)},
howpublished = {The Definitive Guide to AI Data Centers},
year = {2026},
url = {https://aidatacenterguide.com/part-1-strategy-workload-archetypes-and-economics/1-7-the-requirements-and-consequences-matrix},
note = {Accessed 2026-10-05}
}