Inference Data Centers: Bursty, Distributed, Always-On

For most operators, inference is the business. Training builds the asset; inference is what earns against it, request by request, token by token, every hour of every day. In its November 2025 outlook, Deloitte forecasts inference at roughly two-thirds of all AI compute in 2026 — up from about half in 2025 and a third in 2023 (Deloitte TMT Predictions 2026). That is a share of compute, not of installed capacity, energy, or instantaneous power. Yet the reflex in the industry is still to design the inference fleet as if it were a training cluster with the dial turned down. That reflex is expensive. An inference data center optimizes a fundamentally different objective function, and nearly every subsystem decision flips sign because of it.

Chapter 1.1 split the archetypes on one axis: training-shaped facilities optimize a single tightly-coupled job; inference-shaped facilities optimize many independent requests against a latency SLO. This chapter works the consequences of that difference all the way down — the inference taxonomy and the economics shift, the reasoning-driven demand multiplier reshaping the decode-heavy future, latency-driven siting, the prefill/decode disaggregation that has become the defining 2026 serving pattern, the reliability posture, and a GPU-vs-ASIC selection that is genuinely contestable for the first time. The serving engineering — batching, scheduling, the disaggregation tax in detail — is owned by Chapter 10.11; here we cover what those choices mean for the building.

"Inference" covers at least three distinct workloads, and they pull the facility in different directions. The mistake that recurs is treating a fleet as homogeneous when its dominant sub-mode silently dictates the SLO, the prefill:decode ratio, the memory hierarchy, and the fleet size.

Interactive / online inference is a human (or an interactive agent) waiting on the output in real time. It is governed by two latency metrics — time-to-first-token (TTFT), set by the prefill of the prompt, and time-per-output-token (TPOT), set by decode — and it is bursty and always-on: traffic can swing from 30% to 90% of capacity in minutes on a diurnal-plus-spike pattern. The design basis is high availability, proximity to users, and enough headroom to absorb the peak without breaching the SLO.

Batch / offline inference — embeddings generation, document and corpus processing, synthetic-data creation, evaluation sweeps, nightly re-scoring — has no user waiting. It is throughput-bound, not latency-bound, so it tolerates queuing, interruption, and aggressive oversubscription, and it is the natural consumer of spot capacity, off-peak power, and curtailable interconnections. It is the cheapest inference to host and the most flexible to schedule, which makes it the load you shift to soak up the headroom the interactive fleet leaves on the table.

Agentic / long-context inference is the fast-growing third mode and the one that breaks naive sizing. An agent issues many model calls per user action, carries long and growing context (tool outputs, retrieved documents, prior turns reaching toward 1M+ tokens), and interleaves reasoning with tool use. It inflates the prefill share (huge prompts), explodes the KV-cache footprint (long context held live across many concurrent sessions), and shifts the GPU:CPU ratio toward more host work for orchestration and tool calls. A fleet sized on single-turn chat assumptions is undersized for agents on both memory and prefill compute.

That Deloitte share is a forecast of compute, not an observed fleet allocation. Capacity forecasts measure another denominator. McKinsey’s 2025–2030 base case has inference capacity rising from ~20.9 GW to ~93.3 GW by 2030 — a ~35% CAGR — against training's ~23.1 GW to ~62.2 GW at ~22%. The crossover is in motion: training is still the larger installed pool in the base year, but inference compounds ~13 points faster and overtakes it well before 2030 on the same forecast. The consequence for siting and procurement is direct. Training capacity concentrates in a handful of gigawatt campuses chasing cheap firm power; latency-constrained inference capacity distributes toward demand, into more, smaller, latency-sited halls.

The dated price evidence also supplies a downside stress. Ramp's customer cohort paid about $2.50 per million tokens in March 2025, down from about $10 per million roughly one year earlier — a Jevons-paradox dynamic where unit cost collapses while aggregate spend rises because demand more than compensates. That cohort price is not a universal market average. For an inference operator this is a business risk to test against the demand and commitment terms: a fleet scoped to today's $/Mtoken can be underwater in a year if its tokens-per-dollar does not improve on the same curve. So the inference design basis fixates on efficiency per token — tokens-per-watt and tokens-per-dollar at real batch sizes — rather than peak throughput. The full unit-economics build-up and the deflation risk are scored in Chapter 1.8.

Reconcile attempts to originals: 3,600 + 200 = 3,800 attempts; 3,420 + 180 = 3,600 original outcomes. Accepted-original share = 3,420/3,600 = 95%; active/paid GPU-hours = 48/64 = 75%. Neither activity nor an attempt counter proves service attainment. Accepted output = 3,420 × 500 = 1,710,000 tokens; discarded or otherwise unaccepted generation = 2,000,000 − 1,710,000 = 290,000 tokens. Billable output = 3,000 × 500 = 1,500,000 tokens. Accepted output, billable output and generated output are three different denominators.

Let C be the displayed fleet-hour cost and P the displayed receipt per billable accepted request. Cost per million generated tokens is C/2.00, about $50; cost per million accepted output tokens is C/1.71, about $58; cost per million billable output tokens is C/1.50, about $67. Receipts = 3,000P = about $120 for the hour, leaving about $20 over the stated capacity cost before excluded common application costs. Dividing by generated tokens would conceal the retries and rejected output without reducing the bill.

Select HOLD: 95% misses the 97% service requirement even though the hour has positive capacity contribution. The acceptance crossover is ceiling(0.97 × 3,600) = 3,492 distinct accepted originals, 72 more than this trial. Reaching that count at the same quality, latency and paid capacity makes the trial eligible; adding 72 retries without additional accepted originals changes nothing. Independently, contribution changes sign when P = C/3,000, about $0.033 per billable request. Carry the cost boundary, original-request acceptance, useful output and billing rules together to Chapter 1.8; do not extrapolate this hour into annual demand without a traffic ramp. MLCommons’ inference benchmark rules motivate explicit quality, scenario and latency boundaries; production event accounting is canonical in Chapter 14.1.

The table is a cascade, like the one in Chapter 1.1: the left two columns are what you choose; the right-hand columns are hypotheses to qualify. An interactive-dominant fleet must test geo-distribution, a high KV-cache budget, and a willingness to oversubscribe power but not availability. Its revenue-critical service objective often justifies stronger facility topology, but the required maintenance and fault outcomes, not the workload label, select that topology. A batch-dominant fleet can consolidate and accept curtailment when restart and delayed completion still meet its deadline. The agentic column is the one most operators under-provision today, because it looks like interactive chat until the context lengths and call counts reveal a memory-and-prefill problem the single-turn sizing never anticipated.

The largest force reshaping inference demand since 2025 is the rise of reasoning models that spend compute at inference time to improve answers — the post-o1, post-R1 paradigm. A reasoning model emits a long internal chain of thought before its visible answer, turning a request that once decoded a few hundred tokens into one that may decode tens of thousands. This is test-time compute, and it has three structural consequences for the fleet that compound:

• The prefill:decode ratio shifts toward decode. Long chains of thought are autoregressive generation, which is memory-bandwidth-bound, not compute-bound. Decode now dominates the token budget for reasoning traffic, which changes which silicon and which memory hierarchy you want — and tilts the disaggregation math below.
• KV-cache pressure inflates. Every token of context and every token generated must keep its keys and values resident for attention. Long-decode reasoning, multiplied across many concurrent sessions, makes the KV-cache a first-class capacity constraint — often the binding one — not an afterthought. This is the demand driver behind the new inference-memory tier in Chapter 9.7.
• Effective demand per request multiplies. If the average request decodes 10-50x more tokens, a fleet sized on the old token-per-request assumption is undersized by roughly the same factor at constant request volume. Reasoning is, in effect, a demand multiplier hiding inside a flat user count.

Treat the decode-heavy future as the base case rather than a tail risk. It argues for memory-bandwidth-rich silicon, a deep KV-cache hierarchy, and fleet headroom for a per-request token budget that keeps climbing. The serving-engineering levers that absorb this — continuous batching, chunked prefill, speculative decoding — live in Chapter 10.11.

Training siting chases the cheapest firm megawatt and the coldest free-cooling climate, indifferent to where users are. Inference inverts the hierarchy: the binding constraint is the latency budget back to the user, and that turns siting into a coverage problem. Interactive inference must allocate the illustrative 30/50/100 ms service budgets (treated in full for the edge in Chapter 1.5); a centralized fleet whose actual routed propagation and processing exceed 50 ms cannot meet a sub-50 ms latency budget no matter how fast the silicon, because the speed of light in fiber is not negotiable — roughly 0.5 ms per 100 km one-way (about 1 ms round-trip), before any switching or queuing.

The consequence is a different geographic footprint. Where training consolidates into a few gigawatt campuses, inference distributes into more, smaller, latency-sited regions — typically Tier-1 and Tier-2 metros within fiber reach of population centers. Siting closer to users buys proximity and pays for it in energy cost (test a 2-4x metro-versus-rural price scenario against actual delivered tariffs) and in the operational overhead of running many sites instead of one. Site too centrally and you breach the SLO in distant regions; site too widely and you fragment your fleet below the scale where batching efficiency and utilization hold up. The fiber-and-latency screen that governs this is Chapter 3.6; the market-cluster playbook is Chapter 3.13; the reordered siting hierarchy is Chapter 3.1.

The defining serving architecture of 2026 is the recognition that the two phases of a request want different hardware. Prefill — processing the entire prompt to produce the first token and the initial KV-cache — is compute-bound: it saturates the accelerator's matrix engines and scales with prompt length. Decode — generating each subsequent token autoregressively — is memory-bandwidth-bound: each step reads the whole model and the growing KV-cache to emit one token, leaving the matrix engines mostly idle. Run both on the same GPU and they interfere: a long prefill stalls the decode stream of every other request sharing the device, and you cannot tune the hardware for either because it is doing both.

Disaggregated serving splits them onto separate pools — a prefill pool sized for compute, a decode pool sized for memory bandwidth and capacity — connected by a fast fabric that ships the KV-cache from prefill to decode. By early 2026 this is supported across every major open-source engine (vLLM, SGLang, TensorRT-LLM) and orchestrated by frameworks like NVIDIA Dynamo and llm-d, with NIXL the standard KV-transfer transport over RDMA/NVLink. The payoff is higher goodput per dollar and the ability to mix silicon — expensive compute-dense parts for prefill, cheaper memory-rich parts for decode. The cost is the disaggregation tax: every request now pays a KV-cache transfer across the fabric, which only nets out when the pools are large enough and the fabric fast enough that the transfer is cheaper than the interference it eliminates.

The facility consequence is that the KV-cache becomes a first-class, tiered resource — held in HBM where it is hottest, spilled to local NVMe and Ethernet-attached flash as it cools, reused across requests that share a prefix. That memory hierarchy is its own subsystem now, engineered in Chapter 9.7. For the building, disaggregation means the inference hall is no longer a uniform sea of identical nodes; it is a heterogeneous fleet whose pool ratios you tune to your prefill:decode mix — which, per the reasoning shift above, is moving.

Two of inference's defining properties — loose coupling and uncorrelated per-request load — open headroom that a synchronous training job forbids, and capturing it is one of the clearest efficiency levers in the fleet.

Fabric. Many local inference requests fit inside one node or a small scale-up domain, so some designs validate upper-tier 2:1–3:1 oversubscription; a named 2:1-versus-1:1 model estimates roughly 31% lower back-end cost. Those are scoped examples, not an inference default. Distributed MoE, KV movement, or prefill/decode disaggregation can require high bisection. Derive each tier from measured request, KV and expert-parallel traffic, placement, failure headroom, and the tail-latency SLO, then validate it on the target fabric. Topology and oversubscription are engineered in Chapter 8.5.

Power. Training's synchronous steps make thousands of GPUs draw their peak in lockstep, leaving only ~3% power-oversubscription headroom — the facility must provision near the synchronized peak. Inference's per-request peaks are uncorrelated, so the aggregate load is far smoother — ~21% of unused headroom in the production fleets the 2024 study characterized. That is evidence the headroom exists in inference, not an allowance your fleet inherits: derive deployable oversubscription from your own measured coincident demand, the capping and protection response you have actually tested, and the serving degradation you will allow (Uptime Institute Journal; arXiv power-profile studies). The result is density-per-megawatt that training cannot match — but it must be engineered with capping and ride-through, because reasoning and agentic traffic make the per-request peaks lumpier than single-turn chat. The grid-facing transient behavior this implies is in Chapter 12.2's reliability frame.

Training and inference expose different outage consequences, but neither workload label selects a facility topology. Checkpoint-and-resume can reduce a training job’s lost work and recovery cost; request retry, replica capacity, zone routing, and regional failover can mask an inference-site interruption. In both cases, define the maintenance and fault states, allowable transfer interruption, post-event capacity, path and control independence, common modes, recovery SLO, and contractual consequence.

An always-on inference business may place high value on uninterrupted single-site service, but that does not automatically require 2N or Tier-IV-class power: sufficient independent replica, zone, or region capacity can satisfy the end-to-end serving objective with a different site topology. Conversely, a training contract or recovery limit can require path continuity despite checkpointing. Select the site and fleet designs together, then prove the named states. The quantitative comparison is in Chapter 12.5.

Merchant training is dominated by one answer, the rack-scale NVLink domain, though it is not the only one — AMD Instinct systems have carried published end-to-end large-model training programs (AMD/Zyphra, 2025–2026), so the axis to score is software migration cost, demonstrated scale, and time-to-target-quality rather than the absence of an alternative. On the NVIDIA side, GB300 NVL72 shipped in 2025 and is deploying through 2026 alongside GB200; Vera Rubin NVL72 entered full production in August 2026 (NVIDIA Q2 FY27). The workload rewards raw FLOPS, the largest scale-up domain, and the most mature collective-comms software. Inference is the first archetype where the silicon choice is a genuine, contestable fork, because the scoring metric changes from peak FLOPS to tokens-per-watt and tokens-per-dollar at your real batch sizes, context lengths, and SLOs. On that scoreboard, purpose-built inference ASICs become competitive, and several are now in volume. The GPU they are scored against is that same rack: the 72-GPU NVLink domain is what makes prefill/decode disaggregation economic, which is why Azure built its first GB300 cluster for OpenAI reasoning-model inference (October 2025) and AWS's P6e-GB300 instances reached GA in December 2025.

Google's Ironwood (TPU v7) is explicitly "the first TPU for the age of inference": 192 GB of HBM3E at 7.4 TB/s per chip, 4,614 FP8 TFLOPS, scaling to 9,216-chip pods delivering 42.5 FP8 ExaFLOPS — a memory-bandwidth-rich design aimed squarely at the decode-heavy future. AWS's Inferentia/Trainium line and the hyperscaler XPUs (Maia, MTIA) target the same tokens-per-dollar objective for captive fleets. At Cloud Next 2026 (April) Google previewed the eighth-generation split behind it — TPU 8t (training: 3D torus, 9,600-chip superpods, Google-claimed up to 2.7x training $/perf vs Ironwood) and TPU 8i (inference: a new all-to-all 'Boardfly' interconnect, up to 80% better inference $/perf at low-latency MoE serving; both up to 2x perf/W — vendor figures, Apr 2026). Ironwood remains the GA part in 2026. These are surveyed in Chapter 7.4; the qualified configuration from Chapter 7.11 enters the lifecycle cost model in Chapter 1.8.

An ASIC can win decisively on tokens-per-watt for a stable, high-volume model family — but it carries software-ecosystem and lock-in risk (the CUDA moat is real, and a custom part with a thin software stack can leave throughput on the table), and it is a bet that the model architecture you optimize for stays put. Against the Ramp customer cohort's roughly $10/M → $2.50/M decline through March 2025 (not a universal market average) and reasoning reshaping the prefill:decode mix, a part optimized for last year's workload can be the wrong part this year. The rational posture for most operators is to score the fork at their own batch sizes and context lengths — peak-FLOPS datasheets are nearly useless here — and to keep the silicon decision as reversible as the procurement decision, because both move faster than a facility's depreciation clock.

The recurring inference mis-scopes all trace to one root cause: designing the inference fleet as a de-rated training cluster instead of from its own objective function. Four are worth naming:

• Sizing fabric from the word inference. A named 1:1-versus-2:1 model estimates a ~31% back-end cost difference, but the workload label does not set the ratio. Node-local serving may validate upper-tier oversubscription; distributed MoE, KV movement, or prefill/decode disaggregation may require more bisection. Derive each tier from measured traffic, placement, failure headroom, and the tail-latency SLO.
• Sizing the whole fleet to single-turn chat. Reasoning and agentic traffic multiply the per-request token budget and explode KV-cache pressure; a fleet sized on old token-per-request assumptions can exhaust memory or decode capacity, with the factor set by the model, cache policy and admitted concurrency. Replay the declared mix, including long prompts, decode tails and tool waits.
• Centralizing inference for power economics. Consolidating an interactive fleet onto one cheap-power gigawatt campus breaches the latency SLO for users outside its measured path budget. Distribute toward demand and pay the energy premium knowingly.
• Buying peak FLOPS instead of tokens-per-watt. Selecting inference silicon on datasheet FLOPS ignores the memory-bandwidth limit when decode is in that regime; the chip that wins on paper can lose 2x on realized goodput at your batch sizes and context lengths.