Training facility vs Inference facility
An inference data center is not a smaller training cluster. It is a different machine with a different objective, even when it runs the same racks. Training is one synchronous job that wants the largest coherent domain it can buy; inference is millions of independent requests that want low cost per token and, where the latency budget demands it, proximity to users.
| Axis | Training facility | Inference facility |
|---|---|---|
| Load shape | synchronous, dense and checkpointable, with MW-scale swings at optimizer cadence | bursty, diurnal and regional; capacity follows demand curves |
| Siting driver | the cheapest firm power you can energize soonest; latency barely matters | latency to users only where a cheaper distant site misses the latency budget; regional coverage and edge tiers follow from that |
| Scale-up domain | as large as you can buy, because tensor and expert parallelism limits set throughput | sized to the model served; headroom for traffic peaks matters more |
| Storage pressure | checkpoint write bursts and dataset feeds | the KV-cache hierarchy: HBM, then DRAM, then NVMe (Chapter 9.7) |
| Reliability frame | goodput: one failure stalls the whole job, so checkpoint math rules | graceful degradation: lose a replica, not the service; SLOs per request |
| Economics | $/GPU-hr and time to train; utilization is scheduled | $/M tokens and goodput per dollar; utilization is earned request by request |
Quantitative cells are the guide's canonical figures — each is date-stamped and sourced in the numbers register and derived in the chapters below.
How the decision falls
Decide the archetype before the site. Training wants one tightly coupled fabric and the cheapest firm power it can energize soonest; inference wants request-level failover, KV-cache storage tiers, and sites near users only where a cheaper site misses the latency budget. Rack density no longer separates them, because frontier inference runs on the same liquid-cooled NVL72 racks as training (Chapter 1.1). Mixed workloads such as RL and fine-tuning exist; design for that mix on purpose rather than landing in it by compromise.
What would flip it: Long-running agentic inference that behaves like training moves the split away from facility type and toward scheduling policy.
Model this fork with your own numbers: GPU TCO calculator →
Full derivations, worked examples, and the numbers behind this matrix: The archetype decision framework (Ch 1.1) · Training data centers (Ch 1.2) · Inference data centers (Ch 1.3)