Chapter 1.7
In this chapter · 6 sections
The Requirements-and-Consequences Matrix
Name the workload profile, then turn its measured demands into a signed design basis. Cooling, fabric, storage, redundancy, and siting require their own named equipment, facility, traffic, service, and roadmap evidence; no workload label selects them by itself.
What you'll decide here
- The cooling modality each hall is plumbed for — air, rear-door, or direct-to-chip liquid — which the named rack’s heat split, inlet limits and supported interfaces select before steel is cut and which a retrofit cannot cheaply undo.
- The cooling-service envelope for each named rack, plus the back-end blocking ratio and the GPU:CPU, GPU:memory and GPU:storage ratios — starting from the archetype's published pattern and re-set from the rack's heat split and flux, the measured traffic and the service objective.
- The storage tier and its throughput floor (checkpoint write bandwidth, data-loader read bandwidth, KV-cache capacity), mapped to the archetype's tolerance for a stalled GPU.
- The redundancy posture — a restart-tolerant single path only when its interruption is allowed, or no load loss for the declared maintenance and fault states — and the recovery limit that decides which side of that line a fleet falls on.
- Whether the site is scored power-first or latency-first, and the per-archetype reference design-basis sheet you freeze and sign before ordering long-lead equipment.
Chapter 1.1 established the governing input — the workload archetype — and walked the cascade qualitatively. Here it becomes an engineering instrument: a requirements-and-consequences matrix that takes the measured workload, service limits and named equipment and records a concrete, numbered design basis for every subsystem that follows. Where 1.1 established the workload fork, this chapter requires evidence for which inlet temperature, which flow rate, which floor-loading class, which blocking ratio, which storage throughput floor, and which redundancy tier — along with the downstream cost of each cell you fill in wrong.
The altitude here is lower than in 1.1. We move through four mappings in the order an engineer actually commits them — density to the cooling cliff, fabric and the GPU:CPU/memory/storage ratios, storage and redundancy against interruption tolerance, and siting as power-first versus latency-first — and close on the reference design-basis sheets that capture all four per archetype. Read this chapter with Chapter 1.1 open: this is the table its cascade was promising.
Mapping 1 — rack density and heat split to the supported cooling envelope
The first irreversible commitment is the cooling modality, and rack density is the input that narrows it: rear-door heat exchangers (RDHx) carry a typical 30–40 kW rack and past 50 kW with active doors, and above that the named rack's heat split decides whether direct-to-chip liquid is the only path — the overlap between those bands is where the rack's heat flux, airflow and inlet limits, water interfaces and residual room heat make the call. RDHx can extend an existing air architecture by bringing chilled or tempered water to the door. An AALC or in-rack L2A sidecar closes the local liquid loop against room air instead, trading the door water drop for additional room-air heat rejection. Direct-to-chip liquid (DLC) becomes the normal path when the named rack profile and heat flux require liquid capture at the source. An HPE GB200 NVL72 has a 132 kW nominal rack TDP — roughly 115 kW removed by liquid and 17 kW by residual air — which explicitly defines a liquid-plus-residual-air architecture. GB300 NVL72 shipped in 2025 and deploys through 2026 alongside GB200; Lenovo specifies 135 kW TDP, up to 155 kW peak, and a ~90/10 liquid/air split. The ramp does not soften it: VR200 NVL72 entered full production in August 2026; Pegatron specifies 188 kW Max Q / 228 kW Max P; NVIDIA DSX sets a 330 kW cabinet facility design basis, and Rubin Ultra / Kyber remains a ~600 kW roadmap facility planning point on an 800 VDC path. Two racks at the same draw can still require different services — their heat split, flux, airflow limits and water interfaces differ — which is why the matrix below records the evidence, not just the kilowatts.
Where a future profile requires liquid and the inherited hall lacks structure, distribution, isolation, CDU space, or heat rejection, retrofit can cost ~$5–6M/MW and still strand capacity. The engineering lives in Chapter 5.1 through Chapter 5.4.
| Decision input | Evidence to record | What it can rule in or out | Downstream consequence |
|---|---|---|---|
| Named rack thermal profile | Rack kW, component heat flux, liquid/residual heat split, OEM-supported thermal interfaces | Whether source capture is required and how much room-air duty remains | Sets rack, manifold, CDU, airflow, and residual-room loads |
| Qualified room-air envelope | Required airflow and static pressure, inlet range, containment/recirculation, acoustics, fan state, service access | Whether conventional or close-coupled air closes for the named rack | Sets fan power, floor/aisle allocation, controls, and operating margin |
| TCS/FWS and entering-water conditions | Coolant, supply/return limits, design delta-T, flow, pressure drop, CDU approach, isolation and redundancy | Whether RDHx or direct liquid closes at the supported operating point | Sets piping, CDU, pumping, controls, and maintenance boundaries |
| RDHx/AALC qualification | Model-specific capacity at stated air/water temperatures, flow, pressure, fan state and capture target; full residual-room rejection for AALC | Whether the bridge closes without exceeding room-air or water capacity | Adds door/sidecar fan, water, condensate, acoustic, and failure obligations |
| Climate and heat rejection | Design-day ambient, economizer hours, water constraint, rejection approach, heat-reuse duty | Which supply points and rejection systems remain feasible | Sets plant capex, PUE/WUE, water use, and reuse grade |
| Service, redundancy and refresh tail | Isolation, maintainability, fault response, spare strategy, future named rack profiles | Whether today's feasible path remains operable and expandable | Sets irreversible structure, routes, capacity reservations, and stranded-asset risk |
Mapping 2 — fabric sizing and the system ratios
The second mapping is the network, and it has two parts: the blocking ratio of the back-end (scale-out) fabric, set by coupling, and the system composition ratios — GPU:CPU, GPU:memory, GPU:storage — set by the archetype's host-side and data-path demands. Both are decisions where the wrong answer wastes money in opposite directions: over-build and you pay for bandwidth that never carries traffic; under-build and you starve the accelerators you spent the most on.
Coupling sets the blocking ratio. Synchronous training keeps the rail tier non-blocking, because the data-parallel all-reduce crosses it on every step; inference runs 2:1–3:1 at the upper tier in some published designs, because request and KV traffic is bursty and mostly leaf-local; and the tiers above either can run far leaner — Meta's training fabrics carry 7:1 at the top. The named 2:1 model estimates roughly a third lower back-end cost, and Chapter 8.5 gives the three tests — locality, exposed communication, failure headroom — that say whether a given tier may take that saving. What moves a tier back toward 1:1 is wide expert parallelism or prefill/decode disaggregation spilling across racks. → Chapter 8.5 (topology & oversubscription), Chapter 8.4 (protocols).
The composition ratios are archetype-specific and shifting. Training historically ran ~4–8 GPU:1 CPU; agentic inference — with host-side sandbox execution, retrieval, tool calls, and RL rollouts — is pulling that toward ~2:1 and below (TrendForce's projection; Intel describes the same direction), which changes the host BOM and the node power budget. GPU:memory is set by per-GPU HBM (H100 80 GB → B200 180 GB → B300 288 GB → Rubin Ultra ~1 TB at the package level, roadmap) plus host RAM, and inference is increasingly KV-cache-bound rather than weight-bound. GPU:storage is a bandwidth ratio, not a capacity one: it is fixed by checkpoint write speed for training and data-loader read speed for both — which is exactly where Mapping 3 begins.
| Archetype | Back-end blocking ratio | GPU:CPU (host) | Dominant memory pressure | Storage demand profile |
|---|---|---|---|---|
| Pre-training | Typically 1:1 non-blocking, 8-rail fat-tree — confirm against the measured collective traffic | ~8:1 (compute-dense host) | HBM for activations; host RAM for staging | Burst checkpoint writes; high sustained read for data loader |
| Post-training / RL | Disaggregated: tight trainer, tolerant rollout pool | Mixed — more CPU on rollout side | KV-cache on rollouts; HBM on trainer | Rollout reads + trainer checkpoints; staleness-tolerant |
| Online inference | Often 2:1–3:1 oversubscribed — sized to the tail-latency SLO and measured traffic matrix | ~2:1 and below, from a 4–8:1 training-era norm (agentic host work) | KV-cache capacity & bandwidth | Model-weight load; KV-cache tiering to NVMe/CXL |
| Batch inference | Heavily oversubscribed; cost-optimized | Flexible | Throughput over latency; large batches | Throughput reads; no low-latency requirement |
| Edge inference | Minimal (single node / WAN backhaul) | Constrained by appliance | Single-model resident; small KV | Local model store; periodic sync |
Scope & caveats
Select on the named door/rack, air and water conditions, fan state, containment, heat-capture target, residual room heat, climate/rejection, serviceability, redundancy, and future density.
Reference capacity, not a universal ceiling; verify named door/rack, water and air conditions, fan state, containment, and capture target.
Scope & caveats
Exact HPE product profile. Keep this 132 kW / 115 kW liquid / 17 kW air record separate from the OCP MGX Rev. 1.1 reference profile of 120 kW / approximately 102 kW liquid / 18 kW air.
Scope & caveats
Exact QCT reference. The 45 °C liquid-inlet maximum and 65 °C liquid-return maximum are separate limits, not a prescribed 20 K operating rise. Select a supported operating point, approved fluid, liquid heat load, and design ΔT; ASHRAE W45 describes FWS supply capability, not this product's setpoint.
Separate acceptance maxima, not a prescribed 20 K operating rise; do not attribute these limits to HPE without an HPE document that states them.
Scope & caveats
named reference designs and deployments with different traffic, topology, placement and service objectives
Derive oversubscription from the measured traffic matrix, collective/request mix, topology, failure headroom and SLO; validate it on the target fabric.
Scope & caveats
agentic-inference hosts
Scope & caveats
20–40 kW/rack conversion targets; 100 kW+ conversions approach or exceed greenfield cost
Scope & caveats
Secondary-source whole-facility planning estimate quoted by Savills in May 2024; no disclosed estimating population or method. No universal multiplier follows from Uptime Tier criteria.
Single-source planning heuristic: Savills (May 2024) attributes it to Dgtl Infra, which publishes no sample, geography, density or estimating method. The older ~10–25% inverts a McKinsey 2011 statement (10–20% saving moving Tier IV→III). Uptime Tiers are outcome-based, so no universal cost multiplier follows from the standard; re-estimate against the actual design.
Scope & caveats
Stipulated endpoints for sensitivity only. They are neither measured provider outcomes nor universal targets. Chapter 14.1 reconciles productive-time boundaries; a site measures its own baseline.
Mapping 3 — storage and redundancy against interruption tolerance
Storage and redundancy are two consequences of the same input — the archetype's tolerance for an interrupted GPU — and they are most defensible when designed together. The question storage answers is: when does a GPU stall waiting on data, and what does that stall cost? The question redundancy answers is: when a node or a power feed fails, does the workload restart cheaply or lose money?
Storage is sized by the throughput that keeps GPUs fed, not by capacity alone. For training, the two binding flows are checkpoint write bandwidth — because a synchronous job pauses all GPUs to write a checkpoint, and slow writes are pure goodput loss — and data-loader read bandwidth, because a starved loader idles the whole pipeline. A high-bandwidth parallel file system feeding GPUDirect Storage (CPU-bypass) is the training default; this is the link that turns a storage decision into a GPU-efficiency decision (Chapter 9.1, Chapter 9.3, Chapter 9.4). For online inference, the new pressure is the KV-cache: reasoning models emit long decode sequences, inflating per-request cache, so the hierarchy now tiers KV state across HBM, host memory, and NVMe/CXL (Chapter 9.7). Batch and edge are the relaxed cases — throughput reads with no low-latency floor.
Redundancy is set by interruption tolerance, and over-building it is a recognizable waste. A synchronous training job already restarts from a checkpoint when any node fails — SemiAnalysis reported ~7 days of MTBF for one 512-H100 cluster at a top-tier operator, while Meta's separate Llama 3 405B run logged ~one interruption every three hours on 16,384 H100s; these differently defined observations are not one scaling curve — so a single-path or component-redundant posture is viable when checkpoint recovery meets the service objective (Chapter 1.1's over-provisioned-redundancy anti-pattern — stronger topology must earn its cost through a required outcome). An always-on inference business often inverts this: when maintenance or one component fault would breach the SLA and lose revenue, require no load loss for either event. → Chapter 12.1 (redundancy topologies), Chapter 12.2 (goodput vs availability), Chapter 12.4 (goodput SLAs).
| Archetype | Interruption tolerance | Binding storage flow | Storage tier | Redundancy posture |
|---|---|---|---|---|
| Pre-training | High — checkpoint-and-resume | Checkpoint write + loader read bandwidth | Parallel FS + NVMe; GPUDirect Storage | Restart-tolerant; a single path is acceptable when checkpoint recovery meets the objective |
| Post-training / RL | High — staleness-tolerant, restartable | Rollout reads + trainer checkpoints | Tiered: fast trainer FS + rollout object store | Restart-tolerant; separate trainer and rollout fault domains, then size redundancy to recovery limits |
| Online inference | Low — outage = lost revenue + SLA breach | Weight load + KV-cache bandwidth | KV tiered HBM→host→NVMe/CXL | No load loss for maintenance or one component fault when either event breaches the serving SLO |
| Batch inference | High — queue-and-retry | Throughput reads | Object store / capacity tier | Queue-and-retry tolerant; a single path is acceptable when delay stays inside the objective |
| Edge inference | Site-level — fleet geo-redundancy | Local model store + periodic sync | Local NVMe; minimal | Per-site interruption is acceptable when fleet routing preserves the latency objective |
Deep dive: why checkpoint bandwidth and facility redundancy are the same decision
It is tempting to file checkpoint storage under "storage" and redundancy under "electrical," and to size them in separate workstreams. For training, that separation hides the trade. A synchronous job's resilience strategy is checkpoint-and-resume: a recoverable node failure is absorbed by reloading a surviving checkpoint and replaying. The cost of that strategy is two-fold — the goodput lost while all GPUs pause to write each checkpoint, and the work re-done since the last one. Both shrink as checkpoint write bandwidth rises: faster writes mean you can checkpoint more often (less re-done work) at lower per-checkpoint cost (less pause).
So checkpoint bandwidth and facility redundancy are two prices for the same thing — tolerated interruption — and should be bought from one budget: for a checkpointable job, compare faster checkpoints with a second power path against the same allowed interruption and completion budget; Chapter 12.2 ranks the two in one project model, and the recovery limit in the contract is what flips the order. → Chapter 9.4, Chapter 12.2.
Mapping 4 — qualify power-first and latency-first sites
The fourth mapping is the least reversible of all — you cannot move a slab — which is why it must be derived from the workload, never chosen first and rationalized after. Latency sensitivity is the discriminator. Pre-training and batch inference are indifferent to user proximity, so they are scored power-first: chase the cheapest firm (or curtailable) megawatts and the coldest free-cooling climate, accept that the site may be hours from any metro, and treat the grid-interconnection queue slot as the scarcest asset in the project. Online and edge inference are scored latency-first: chase sub-50 ms reach to users and accept power that can cost 2–4x more, distributing capacity for proximity rather than concentrating it for cost.
The 2026 context sharpens this fork. The binding constraint is often power, not chips. Because large-load service dates are utility-, tariff-, study-, upgrade- and project-specific, a power-first archetype that mis-sites near expensive, constrained metro power burns both money and a queue slot it cannot recover. A latency-first archetype sited in a cheap-power exurb, conversely, may meet its energy budget and miss its SLO, which is the more expensive miss because it loses the revenue the building exists to earn. Facility water is a siting gate for water-dependent cooling topologies; where it is unavailable, choose liquid-to-air CDUs or closed-loop dry heat rejection if the ambient envelope closes (Chapter 3.7). The reordered hierarchy and the speed-to-power race are engineered in Chapter 3.1 and Chapter 3.2; the fiber/latency screen in Chapter 3.6.
The reference design-basis sheet, per archetype
The four mappings converge into a single artifact: a reference design-basis sheet per materially different workload and equipment profile that freezes the accepted assumptions before any long-lead equipment is ordered. This is the deliverable 1.1 promised under "design-basis document," made explicit in the illustrative record below. Each sheet pins one row per subsystem — density tier, cooling modality, fabric blocking ratio, system ratios, storage tier and throughput floor, redundancy topology, voltage class, and siting class — plus a reversible-vs-irreversible register recording which assumptions are hedged and which are committed. The table below is the skeleton; a real sheet attaches the numbers (the ramp curve, the MVA sizing, the CDU capacity) and the acceptance evidence, owner and approval state.
| Subsystem | Pre-training | Frontier inference (NVL72-class) | Enterprise inference (8-GPU node) | Batch inference | Edge inference |
|---|---|---|---|---|---|
| Density tier | 100–600 kW (DLC) | Lenovo GB300 NVL72: 135 kW TDP / up to 155 kW peak; NVIDIA RA facility basis up to 142 kW; Pegatron VR200: 188 kW Max Q / 228 kW Max P; NVIDIA DSX facility design basis: 330 kW | 14.5 kW consumption / 15 kW max per DGX B300 node | 30–60 kW (flexible) | 5–50 kW networked edge DC class |
| Cooling modality | DLC, warm-water loop | DLC, warm-water loop | Air or DLC per OEM system and site envelope | Host hall's existing (air often fine) | Air / sealed modular |
| Fabric blocking | Typically 1:1 non-blocking (confirm on measured collectives) | Sized to the tail-latency SLO and the measured traffic matrix, not to the workload label | Often 2:1–3:1 (SLO- and traffic-sized) | Heavily oversubscribed | Minimal / WAN backhaul |
| Storage tier | Parallel FS + GPUDirect | KV-tiered HBM→host→NVMe/CXL | KV-tiered HBM→NVMe | Object / capacity tier | Local NVMe |
| Redundancy | Restart-tolerant; single path acceptable if recovery meets the objective | No load loss for maintenance or one fault when the SLA requires it | No load loss when the SLA requires it; otherwise fleet routing absorbs a node loss | Queue-and-retry tolerant; single path acceptable | Per-site interruption acceptable if fleet routing preserves latency |
| Voltage class | 415/480 VAC → 800 VDC path | 415/480 VAC → 800 VDC path | 415/480 VAC | 415/480 VAC | Local LV / appliance |
| Siting class | Power-first (cheap, cold, big queue) | Power-first or metro-anchored campus | Latency-first (sub-50 ms) | Cheapest / curtailable power | Proximity over cost |
A filled design basis for a small serving fleet
Close the normal and failed states: eight nodes × eight GPUs = 64 installed GPUs. A node holds 70 × 10⁹ × 2 = 140 GB of weights; weights + KV + buffers = 140 + 120 + 20 = 280 GB against 8 × 80 = 640 GB of aggregate HBM. This passes the aggregate screen; tensor placement, temporary peaks and the 8,000-token cap still require a runtime test. Installed host capacity is 16 CPU sockets, 4,096 GiB RAM and 16 TB local staging. Storage acceptance must demonstrate model loading, restart and request-log durability within the recovery requirement; local staging capacity is not a durable checkpoint guarantee.
Peak offered attempts = 10 × 1.10 = 11 attempts/s. Healthy candidate throughput is 8 × 2 = 16 attempts/s; with one node unavailable it is 7 × 2 = 14 attempts/s, leaving 3 attempts/s of headroom. At the assumed 1 MB/attempt, peak external traffic is approximately 10 MB/s (0.09 Gb/s), below one surviving 100 Gb/s attachment; this screen does not qualify KV transfers, model distribution or tail behavior. Admit no cross-node model partition in this profile without reopening the fabric basis.
IT demand = 8 × 12 + 4 = 100 kW, or 25 kW/rack against the assumed 30 kW rack envelope. One cooling unit out leaves 2 × 60 = 120 kW against 100 kW of IT heat. Coincident facility demand = 100 + 28 = 128 kW and apparent demand = 128/0.95, about 1.3 × 10² kVA; one surviving 160 kW/200 kVA path passes both screens. An independent path must reach the rack and its controls. The inlet, airflow, heat-rejection state and safe isolation route remain part of the acceptance record; an annual PUE cannot qualify any of them.
Select the eight-node candidate for qualification, with procurement on HOLD. The service owner must produce the offered trace and quality evaluation; the platform supplier must prove memory placement, node rate and supported rack envelope; the facility engineer must prove power, cooling, floor load and delivery route; operations must demonstrate drain, retry, fault detection and restoration. Release follows their evidence, not the arithmetic alone. The CPU and staging assumptions are ceilings to test, not ratios that the workload label guarantees.
Flip: with one node unavailable, the node-rate crossover is 11/7, about 1.6 attempts/s. At 1.5 attempts/s, seven survivors deliver 10.5 attempts/s and miss demand by 0.5. Nine installed nodes leave eight survivors and 12 attempts/s, but the extra 12 kW node also requires a new placement: a third node would push one existing rack from 25 to 37 kW, beyond its 30 kW envelope. Choose an additional qualified rack, lower admitted demand or a faster qualified configuration; do not solve serving capacity by overrunning the cooling basis. MLCommons supplies the primary quality-and-latency qualification pattern. The method homes are Chapter 7.6 for memory, Chapter 5.1 for heat, Chapter 8.5 for fabric and Chapter 13.5 for integrated acceptance; Chapter 1.8 prices the resulting paid fleet.
Deep dive: reading the matrix backwards to audit an existing facility
The matrix is written forward — archetype in, design basis out — but its most useful diagnostic mode is backward. Given a facility that already exists (a hall you are evaluating to lease, retrofit, or acquire), read its observable subsystems back up the cascade and infer the archetype it was actually built for, then compare that to the workload you intend to run.
A hall whose named 40 kW rack profile closes under recorded airflow and inlet conditions, with an oversubscribed Ethernet fabric, duplicated power, and a metro location may suit a latency-sensitive serving objective; a different training rack may fail that inherited thermal envelope, starve the all-reduce, and pay for redundancy beyond its recovery requirement. A campus with NVL72-class DLC racks and a non-blocking InfiniBand fabric no longer identifies its own workload: frontier serving deploys the same racks, and disaggregated prefill/decode plus MoE all-to-all consume real bisection. The discriminators sit elsewhere — fabric scale and the blocking ratio per tier, whether storage is sized for checkpoint bursts or for KV-cache tiering, and whether the failure model assumes restart-from-checkpoint or live request retry. Component redundancy and a remote cheap-power site still read as restart-tolerant compute; the same racks in a metro hall that tolerates no load loss for a single fault read the other way. The mismatches the backward read exposes are exactly the three anti-patterns 1.1 named: training fabric for an inference business, forcing a named rack beyond the inherited cooling envelope, and over-provisioned redundancy for checkpointable jobs. The matrix is therefore both a scoping tool and a due-diligence checklist — the same table, run in two directions. → Chapter 5.10 (retrofit limits), Chapter 1.6 (procurement diligence).
Cite this chapter
Fehn, J. (2026). The Requirements-and-Consequences Matrix (Chapter 1.7). The Definitive Guide to AI Data Centers. https://aidatacenterguide.com/part-1-strategy-workload-archetypes-and-economics/1-7-the-requirements-and-consequences-matrix (accessed 2026-09-29).
@misc{aidc-1-7,
author = {Fehn, Jacob},
title = {The Requirements-and-Consequences Matrix (Chapter 1.7)},
howpublished = {The Definitive Guide to AI Data Centers},
year = {2026},
url = {https://aidatacenterguide.com/part-1-strategy-workload-archetypes-and-economics/1-7-the-requirements-and-consequences-matrix},
note = {Accessed 2026-09-29}
}