Inference Data Centers: Bursty, Distributed, Always-On

For most operators, inference is the business. Training builds the asset; inference earns against it, request by request and token by token, every hour of every day. In its November 2025 outlook, Deloitte forecasts inference at roughly two-thirds of all AI compute in 2026, up from about half in 2025 and a third in 2023 (Deloitte TMT Predictions 2026). That is a share of compute, not of installed capacity, energy or instantaneous power. Yet the industry's reflex is still to design the inference fleet as a training cluster with the dial turned down, and that reflex is expensive. Inference optimizes a different objective, and nearly every subsystem decision flips sign because of it.

Chapter 1.1 split the archetypes on one axis: training-shaped facilities optimize a single tightly coupled job, while inference-shaped facilities optimize many independent requests against a latency SLO. This chapter works through the consequences: the inference taxonomy and its economics, the reasoning-driven demand multiplier, latency-driven siting, prefill/decode disaggregation (the defining 2026 serving pattern), the reliability posture, and a GPU-versus-ASIC choice that is a real contest for the first time. The serving engineering (batching, scheduling, the disaggregation tax in detail) belongs to Chapter 10.11. Here we cover what those choices mean for the building.

"Inference" covers at least three workloads, and they pull the facility in different directions. The recurring mistake is treating a fleet as homogeneous when its dominant sub-mode dictates the SLO, the prefill:decode ratio, the memory hierarchy and the fleet size.

Interactive / online inference has a human (or an interactive agent) waiting on the output in real time. Two latency metrics govern it: time-to-first-token (TTFT), set by the prompt's prefill, and time-per-output-token (TPOT), set by decode. It is bursty and always on: traffic follows a diurnal curve with launch- and news-driven spikes on top, so take the peak-to-trough ratio from your own request logs. The design basis is high availability, proximity to users, and enough headroom to absorb the peak without breaching the SLO.

Batch / offline inference covers embeddings generation, document and corpus processing, synthetic-data creation, evaluation sweeps and nightly re-scoring. No user is waiting, so it is bound by throughput, not latency. It tolerates queuing, interruption and aggressive oversubscription, and it is the natural consumer of spot capacity, off-peak power and curtailable interconnections. It is the cheapest inference to host and the most flexible to schedule, which makes it the load you shift in to soak up the headroom the interactive fleet leaves on the table.

Agentic / long-context inference is the fast-growing third mode and the one that breaks naive sizing. An agent issues many model calls per user action, carries long and growing context (tool outputs, retrieved documents, prior turns retained within the model’s context window) and interleaves reasoning with tool use. That inflates the prefill share with huge prompts, explodes the KV-cache footprint because long context stays live across many concurrent sessions, and shifts the GPU:CPU ratio toward more host work for orchestration and tool calls. A fleet sized on single-turn chat is undersized for agents on both memory and prefill compute.

That Deloitte share is a forecast of compute, not an observed fleet allocation. Capacity forecasts use another denominator. McKinsey’s 2025–2030 base case has inference capacity rising from ~20.9 GW to ~93.3 GW by 2030, a ~35% CAGR, against training's ~23.1 GW to ~62.2 GW at ~22%. The crossover is under way: training is still the larger installed pool in the base year, but inference compounds ~13 points faster and overtakes it well before 2030 on the same forecast. The consequence for siting and procurement is direct. Training capacity concentrates in a handful of gigawatt campuses chasing cheap firm power; latency-constrained inference capacity distributes toward demand, into more, smaller, latency-sited halls.

Falling prices supply a downside stress. Ramp’s 9 September 2026 update puts its effective token-price index at $0.68 per million tokens, 41% below the March 2026 peak of $1.15. The index tracks prices and model mix among businesses Ramp observes, not a universal tariff. Stress realized receipts separately from token volume; cheaper tokens do not guarantee higher spending. For an inference operator this is a business risk to test against demand and commitment terms: a fleet scoped to today's $/Mtoken can be underwater in a year if its tokens-per-dollar does not improve on the same curve. So the design basis fixates on efficiency per token, meaning tokens-per-watt and tokens-per-dollar at real batch sizes, rather than peak throughput. Chapter 1.8 scores the full unit-economics build-up and the deflation risk.

Reconcile attempts to originals: 3,600 + 200 = 3,800 attempts; 3,420 + 180 = 3,600 original outcomes. Accepted-original share = 3,420/3,600 = 95%; active/paid GPU-hours = 48/64 = 75%. Neither activity nor an attempt counter proves service attainment. Accepted output = 3,420 × 500 = 1,710,000 tokens; discarded or otherwise unaccepted generation = 2,000,000 − 1,710,000 = 290,000 tokens. Billable output = 3,000 × 500 = 1,500,000 tokens. Accepted output, billable output and generated output are three different denominators.

Let C be the displayed fleet-hour cost and P the displayed receipt per billable accepted request. Cost per million generated tokens is C/2.00, about $50; cost per million accepted output tokens is C/1.71, about $58; cost per million billable output tokens is C/1.50, about $67. Receipts = 3,000P = about $120 for the hour, leaving about $20 over the assumed capacity cost before excluded common application costs. Dividing by generated tokens would conceal the retries and rejected output without reducing the bill.

Select HOLD: 95% misses the 97% requirement even though the hour has positive capacity contribution. The acceptance crossover is ceiling(0.97 × 3,600) = 3,492 distinct accepted originals, 72 more than this trial. Reaching that count at the same quality, latency and paid capacity makes the trial eligible; adding 72 retries without additional accepted originals changes nothing. Independently, contribution changes sign when P = C/3,000, about $0.033 per billable request. Carry the cost boundary, original-request acceptance, useful output and billing rules together to Chapter 1.8, and do not extrapolate this hour into annual demand without a traffic ramp. MLCommons’ inference benchmark rules motivate explicit quality, scenario and latency boundaries. Production event accounting is canonical in Chapter 14.1.

The table is a cascade, like the one in Chapter 1.1: the left two columns are what you choose, and the right-hand columns are hypotheses to test. An interactive-dominant fleet must test geo-distribution, a high KV-cache budget, and a willingness to oversubscribe power but not availability. Its revenue-critical service target often justifies a stronger facility topology, but the maintenance and fault outcomes you require, not the workload label, select that topology. A batch-dominant fleet can consolidate and accept curtailment when restart and delayed completion still meet its deadline. The agentic row is the one most operators under-provision today: it looks like interactive chat until context lengths and call counts reveal a memory-and-prefill problem that single-turn sizing never anticipated.

The largest force reshaping inference demand since 2025 is the rise of reasoning models that spend compute at inference time to improve answers, the post-o1, post-R1 paradigm. A reasoning model emits a long internal chain of thought before its visible answer, turning a request that once decoded a few hundred tokens into one that may decode tens of thousands. This is test-time compute, and it has three consequences for the fleet that compound:

• The prefill:decode ratio shifts toward decode. Long chains of thought increase autoregressive decode work; the bottleneck depends on batch size, context and model. Decode now dominates the token budget for reasoning traffic, which changes the silicon and memory hierarchy you want and tilts the disaggregation math below.
• KV-cache pressure inflates. Full-attention layers retain KV state across the attended context; sliding-window layers need only their active window. Long-decode reasoning, multiplied across many concurrent sessions, makes the KV-cache a first-class capacity constraint, often the binding one, and it drives the new inference-memory tier in Chapter 9.7. Size the live KV budget from the model’s attention layout, cache precision, retained context and admitted concurrency.
• Effective demand per request multiplies. If the average request decodes 10-50x more tokens, a fleet sized on the old token-per-request assumption is undersized by roughly the same factor at constant request volume. Reasoning is a demand multiplier hiding inside a flat user count.

Treat the decode-heavy future as the base case, not a tail risk. It argues for memory-bandwidth-rich silicon, a deep KV-cache hierarchy and fleet headroom for a per-request token budget that keeps climbing. The serving levers that absorb it (continuous batching, chunked prefill, speculative decoding) live in Chapter 10.11.

Training siting chases the cheapest firm megawatt and the coldest free-cooling climate, indifferent to where users are. Inference inverts the hierarchy: the binding constraint is the latency budget back to the user, which turns siting into a coverage problem. Interactive inference must allocate the illustrative 30/50/100 ms service budgets (treated in full for the edge in Chapter 1.5). A centralized fleet whose actual routed propagation and processing exceed 50 ms cannot meet a sub-50 ms budget however fast the silicon, because the speed of light in fiber is not negotiable: roughly 0.5 ms per 100 km one-way (about 1 ms round-trip), before any switching or queuing.

The result is a different footprint. Where training consolidates into a few gigawatt campuses, inference distributes into more, smaller, latency-sited regions, mostly Tier-1 and Tier-2 metros within fiber reach of population centers. Proximity to users costs energy and operations. Metro power typically costs about 1.5–2x the cheapest US industrial rates (EIA, July 2026), and 2.5x or more only in California- or New Jersey-type markets, so test your own site pair against actual delivered tariffs. Running many sites instead of one adds operational overhead. Site too centrally and you breach the SLO in distant regions; site too widely and you fragment the fleet below the scale where batching efficiency and utilization hold up. Chapter 3.6 covers the fiber-and-latency screen, Chapter 3.13 the market-cluster playbook, and Chapter 3.1 the reordered siting hierarchy.

The defining serving architecture of 2026 recognizes that the two phases of a request want different hardware. Prefill processes uncached prompt tokens and builds the initial KV-cache; it commonly becomes compute-bound when the prompt or batch supplies enough parallel work. Decode generates each subsequent token and is typically memory-bandwidth-bound at small batches, because each step reads the whole model and the growing KV-cache to emit one token; larger batches can make its linear operators compute-bound. On one shared GPU, a long prefill can stall the decode streams of other requests, and you cannot tune the hardware for either phase. Chunked prefill trades first-token latency against decode cadence. Choose the pools from the target model’s measured prompt, batch and context distribution; Chapter 10.11 owns the serving mechanics.

Disaggregated serving splits the phases onto separate pools: a prefill pool sized for compute and a decode pool sized for memory bandwidth and capacity, connected by a fast fabric that ships the KV-cache from prefill to decode. By early 2026 every major open-source engine supports this (vLLM, SGLang, TensorRT-LLM), frameworks like NVIDIA Dynamo and llm-d orchestrate it, and NIXL is the standard KV-transfer transport over RDMA/NVLink. The payoff is higher goodput per dollar and the ability to mix silicon: expensive compute-dense parts for prefill, cheaper memory-rich parts for decode. The cost is the disaggregation tax. Every request now pays a KV-cache transfer across the fabric, which only nets out when the pools are large enough and the fabric fast enough that the transfer is cheaper than the interference it eliminates.

For the facility, the KV-cache becomes a first-class, tiered resource: held in HBM where it is hottest, spilled to local NVMe and Ethernet-attached flash as it cools, and reused across requests that share a prefix. That memory hierarchy is its own subsystem now, engineered in Chapter 9.7. For the building, disaggregation means the inference hall is no longer a uniform sea of identical nodes. It is a heterogeneous fleet whose pool ratios you tune to your prefill:decode mix, which, per the reasoning shift above, is moving.

Inference's loose coupling and uncorrelated per-request load open headroom that a synchronous training job forbids. Capturing it is one of the clearest efficiency levers in the fleet.

Fabric. Many local inference requests fit inside one node or a small scale-up domain, so some designs validate upper-tier oversubscription. SemiAnalysis models 31.6% lower back-end network cost for a 1,024-H100 training cluster that combines 2:1 leaf-to-spine oversubscription with a copper virtual-modular-switch layout, relative to NVIDIA’s reference architecture (3 October 2024). That is a training-cluster example, not an inference default. Distributed MoE, KV movement or prefill/decode disaggregation can require high bisection. Derive each tier from measured request, KV and expert-parallel traffic, placement, failure headroom and the tail-latency SLO, then validate it on the target fabric. Chapter 8.5 engineers topology and oversubscription.

Power. Training's synchronous steps make thousands of GPUs draw their peak in lockstep, leaving only ~3% power-oversubscription headroom, so the facility must provision near the synchronized peak. Inference's per-request peaks are uncorrelated, so the aggregate load is far smoother: ~21% of unused headroom in the production fleets the 2024 study characterized. That shows the headroom exists in inference; it is not an allowance your fleet inherits. Derive deployable oversubscription from your own measured coincident demand, the capping and protection response you have tested, and the serving degradation you will allow (Uptime Institute Journal; arXiv power-profile studies). Inference can reach more density per megawatt than a synchronized training job, but only with capping and ride-through engineered in, because reasoning and agentic traffic make per-request peaks lumpier than single-turn chat. Chapter 12.2's reliability frame covers the grid-facing transient behavior this implies.

Training and inference have different outage consequences, but neither workload label selects a facility topology. Checkpoint-and-resume can reduce a training job’s lost work and recovery cost. Request retry, replica capacity, zone routing and regional failover can mask an inference-site interruption. For both, define the maintenance and fault states, allowable transfer interruption, post-event capacity, path and control independence, common modes, recovery SLO and contractual consequence.

An always-on inference business may place high value on uninterrupted single-site service, but that does not automatically require 2N or Tier-IV-class power. Enough independent replica, zone or region capacity can meet the end-to-end serving target with a different site topology. Conversely, a training contract or recovery limit can require path continuity despite checkpointing. Choose the site and fleet designs together, then test every defined maintenance and fault state. The quantitative comparison is in Chapter 12.5.

Merchant training is dominated by one answer, the rack-scale NVLink domain, though it is not the only one: AMD Instinct systems have carried published end-to-end large-model training programs (AMD/Zyphra, 2025–2026). The axis to score is software migration cost, demonstrated scale and time-to-target-quality, not the absence of an alternative. On the NVIDIA side, GB300 NVL72 shipped in 2025 and is deploying through 2026 alongside GB200, and Vera Rubin NVL72 racks have shipped since June 2026 (Dell's first racks went to CoreWeave). Training rewards raw FLOPS, the largest scale-up domain and the most mature collective-comms software. Inference is the first archetype where the silicon choice is a real, contestable fork, because the scoring metric changes from peak FLOPS to tokens-per-watt and tokens-per-dollar at your real batch sizes, context lengths and SLOs. On that scoreboard, purpose-built inference ASICs become competitive, and several are now in volume. The GPU they are scored against is that same rack: the 72-GPU NVLink domain is what makes prefill/decode disaggregation economic, which is why Azure built its first GB300 cluster for OpenAI reasoning-model inference (October 2025) and AWS's P6e-GB300 instances reached GA in December 2025.

Google's Ironwood (TPU v7) is explicitly "the first TPU for the age of inference": 192 GB of HBM3E at 7.4 TB/s per chip, 4,614 FP8 TFLOPS, scaling to 9,216-chip pods delivering 42.5 FP8 ExaFLOPS. It is a memory-bandwidth-rich design aimed squarely at the decode-heavy future. AWS's Inferentia/Trainium line and the hyperscaler XPUs (Maia, MTIA) target the same tokens-per-dollar objective for captive fleets. At Cloud Next 2026 (April) Google previewed the eighth-generation split behind it: TPU 8t (training: 3D torus, 9,600-chip superpods, Google-claimed up to 2.7x training performance per dollar versus Ironwood) and TPU 8i (inference: a new all-to-all 'Boardfly' interconnect, Google-claimed up to 80% better inference performance per dollar versus Ironwood at low-latency targets for large MoE models; both up to 2x perf/W; vendor figures, Apr 2026). Ironwood remains the GA part in 2026. Chapter 7.4 surveys these. The configuration you validate with Chapter 7.11 enters the lifecycle cost model in Chapter 1.8.

An ASIC can win decisively on tokens-per-watt for a stable, high-volume model family. But it carries software-ecosystem and lock-in risk (the CUDA moat is real, and a custom part with a thin software stack can leave throughput on the table), and it bets that the model architecture you optimize for stays put. Against Ramp’s effective token-price index falling from $1.15 to $0.68 per million tokens between March and September 2026 (businesses Ramp observes, not a universal tariff), and with reasoning reshaping the prefill:decode mix, a part optimized for last year's workload can be the wrong part this year. The rational posture for most operators is to score the fork at their own batch sizes and context lengths, since peak-FLOPS datasheets are nearly useless here, and to keep the silicon decision as reversible as the procurement decision, because both move faster than a facility's depreciation clock.

The recurring inference mis-scopes trace to one root cause: designing the inference fleet as a de-rated training cluster instead of from its own objective. Four are worth naming:

• Sizing fabric from the word "inference". One published training-cluster model estimates 31.6% lower back-end network cost from 2:1 oversubscription plus a copper virtual-modular-switch layout, but the workload label does not set the ratio. Node-local serving may validate upper-tier oversubscription; distributed MoE, KV movement or prefill/decode disaggregation may require more bisection. Derive each tier from measured traffic, placement, failure headroom and the tail-latency SLO.
• Sizing the whole fleet to single-turn chat. Reasoning and agentic traffic multiply the per-request token budget and explode KV-cache pressure. A fleet sized on old token-per-request assumptions can exhaust memory or decode capacity, with the factor set by the model, cache policy and admitted concurrency. Replay your assumed mix, including long prompts, decode tails and tool waits.
• Centralizing inference for power economics. Consolidating an interactive fleet onto one cheap-power gigawatt campus breaches the latency SLO for users outside its measured path budget. Distribute toward demand and pay the energy premium knowingly.
• Buying peak FLOPS instead of tokens-per-watt. Selecting inference silicon on datasheet FLOPS ignores the memory-bandwidth limit when decode is in that regime. The chip that wins on paper can lose 2x on realized goodput at your batch sizes and context lengths.