Compared › Training facility vs Inference facility
Training facility vs Inference facility
An inference data center is not a smaller training cluster — it is a different machine optimized for a different objective. Training is one synchronous job that wants the biggest possible coherent domain; inference is millions of independent requests that want to be close to users and cheap per token.
| Axis | Training facility | Inference facility |
|---|---|---|
| Load shape | synchronous, dense, checkpointable — MW-scale steps at optimizer cadence | bursty, diurnal, regional — capacity follows demand curves |
| Siting driver | cheapest firm power wins; latency barely matters | latency to users and regional coverage; edge tiers appear |
| Scale-up domain | as big as buyable — TP/EP ceilings set throughput | sized to the model served; oversubscription headroom matters more |
| Storage pressure | checkpoint bandwidth (write bursts), dataset feeds | KV-cache hierarchy — HBM → DRAM → NVMe (Ch 9.7) |
| Reliability frame | goodput: one failure stalls the whole job; checkpoint math rules | graceful degradation: lose a replica, not the service; SLOs per request |
| Economics | $/GPU-hr and time-to-train; utilization is scheduled | $/M tokens and goodput-per-dollar; utilization is earned per request |
How the decision falls
Decide the archetype before the site: a training campus and an inference estate disagree about siting, network, storage, and even what 'reliable' means. The hybrid middle (RL, fine-tuning) exists — but design it consciously as a mix, not accidentally as a compromise.
Full derivations, worked examples, and the numbers behind this matrix: Training data centers (Ch 1.2) · Inference data centers (Ch 1.3) · The archetype decision framework (Ch 1.1)