Host CPUs, GPU:CPU Ratios & System Composition

An accelerator does not run a workload by itself. It runs as the muscle of a system — a host CPU that loads the model, drives the data pipeline, schedules kernels, services interrupts, runs the OS and the container runtime, terminates the storage and management network, and increasingly does real application work: tokenization, retrieval, sandboxed tool execution, agent orchestration, and the reward-evaluation loop of reinforcement learning. The question this chapter answers is not "which GPU" — that is Chapter 7.2 through Chapter 7.5. It is the question that sits one level up: what machine do you wrap around the accelerator, and in what ratio. Get it wrong in one direction and the host starves the GPUs — they idle waiting on data that the CPU pipeline cannot deliver, and you have paid for the most expensive silicon in the building to wait. Get it wrong in the other direction and you have bought host cores and host memory that no kernel ever touches — dead capex riding on a power budget you are otherwise fighting to conserve.

This was a quiet, settled question for the training era. The answer was "as little CPU as you can get away with": eight GPUs per host, the cheapest x86 that could keep the pipeline fed, and as much of the silicon budget as possible spent on accelerators. Two forces broke that settlement. First, coherent host-attach — NVIDIA's NVLink-C2C welding a Grace (and now Vera) Arm CPU to the GPU as a single memory-coherent superchip — turned the host from a discrete component into part of the accelerator's memory hierarchy. Second, and with wider consequences, agentic inference and RL shifted real, sustained compute back onto the host. The orchestration layer that schedules sub-agents, routes tool calls, runs retrieval, and decides whether a task is done is CPU work, and it does not fit in the slivers of host left over from an 8:1 training box. The industry is re-deciding system composition in real time, and the fork has consequences that propagate into power, density, supply, and serviceability for the life of the fleet.

There are two physically and architecturally distinct ways to attach a host CPU to accelerators, and the choice is the first fork in system composition. They differ in the interconnect, the memory model the programmer sees, the failure blast radius, and — critically in 2026 — in how tightly they couple you to a single vendor's supply chain.

Coherent host-attach (the superchip model). NVIDIA's Grace-Blackwell and Vera-Rubin packages connect an Arm host CPU to the GPU(s) over NVLink-C2C, a chip-to-chip link delivering ~900 GB/s of coherent bandwidth on Grace-class parts (rising on Vera). The word coherent is doing real work here: the CPU's LPDDR5X and the GPU's HBM live in one cache-coherent address space, so the GPU can reach host memory and the CPU can reach HBM with hardware coherence, no explicit copy, no staging through a bounce buffer. The host memory becomes a fast, large NUMA tier behind the GPU — the natural home for KV-cache spillover, large embedding tables, MoE expert weights that do not fit in HBM, and the working set of long-context and agentic inference. The cost: the CPU is welded to the GPU. You take the ratio the package ships (2 GPUs : 1 Grace in GB200; the Vera-Rubin pairing in 2026), the LPDDR5X capacity is what the package ships (soldered on Grace, field-replaceable SOCAMM modules on Vera), and you are buying one vendor's CPU, GPU, and interconnect as an indivisible unit.

Discrete host-attach (the PCIe model). The classic HGX/8-GPU baseboard hangs accelerators off a pair of x86 sockets (AMD EPYC or Intel Xeon) over PCIe — Gen5 at ~64 GB/s per direction per x16 (~128 GB/s bidirectional), Gen6 at ~128 GB/s per direction. The CPU and GPU keep separate memory spaces; moving data between them uses DMA or a supported unified-memory/page-migration path over PCIe or, for GPU-to-GPU, a sideband over NVLink that bypasses the host entirely. You give up coherent host memory — host RAM is not a transparent extension of HBM, and PCIe bandwidth is an order of magnitude below NVLink-C2C — but you gain composability. You choose the CPU vendor, the core count, the socketed DDR5 capacity (terabytes if you want them), and the GPU:CPU ratio independently, and you can service or upgrade the host without touching the accelerator. For workloads that do not need coherent host memory — most discrete-GPU inference, and any fleet that values dual-source supply over the last increment of host-GPU bandwidth — this is the pragmatic default.

The GPU:CPU ratio is the number that most concisely captures system composition. A common training host carried eight accelerators and two CPU sockets/packages: eight accelerators per host, or 4:1 when the denominator is CPU packages. Ratios must name their denominator. In this example, the CPU's job was to keep the data pipeline full and stay out of the way. The host did almost no application compute — pre-training is GPU-bound on collectives and matmul, and the CPU's contribution is loading shards, augmenting data, and launching kernels. The rational move was to minimize host spend: the dollars and the watts belonged to the GPUs. Even host memory got cut — neocloud reference builds trimmed training-server RAM from 2 TB to 1 TB precisely because the trainer never used it — a host-BOM option alongside a smaller CPU and removing two BlueField-3 DPUs. SemiAnalysis’s October 3, 2024 example priced those changes at $270k → $256.4k per H100 server, a $13.6k saving in its 1,024-GPU scenario. A new quote must preserve any required DPU isolation or networking function.

Agentic inference and RL break the constant. The work that defines an AI agent — decomposing a request into sub-tasks, scheduling them, routing tool calls, passing state between sub-agents, running retrieval, executing code in sandboxes, and evaluating whether the original goal is met — is overwhelmingly host-side, control-heavy compute. It is branchy, latency-sensitive, and does not vectorize onto a GPU. RL training compounds this: every action an agent takes during a rollout has to be executed and scored, and that evaluation loop runs on the CPU. As agents proliferate, the host stops being a feeder and becomes a co-processor doing sustained work, and the ratio that kept GPUs the scarce resource per node no longer holds. TrendForce projects the ratio shifting from the training-era 4:1–8:1 toward lower accelerator-to-CPU-package ratios in some projected agentic deployments; Intel describes a similar directional shift, but vendor projections require a named denominator and workload profile. Arm goes further on the aggregate: as agents proliferate, orchestration, tool execution and state handling consume host cores while the GPU waits on them. Arm’s addressable-market projection does not specify this rack’s CPU utilization, node power or admission policy.

That rebalance has a concrete consequence. A fleet scoped at 8:1 for training and then repurposed for agentic serving will be CPU-starved: the orchestration and tool-execution layer saturates the thin host, queues form ahead of the GPUs, tail latency blows past the SLO, and accelerators sit underutilized behind a host bottleneck. The fix is not more GPUs — it is more CPU per GPU, which means a different node, often a different rack SKU, and a host-side power and memory budget the training composition never planned for. This is why NVIDIA designed Vera explicitly "for agents": 88 custom Olympus cores, 176 threads via spatial multithreading, and up to 1.5 TB of LPDDR5X at ~1.2 TB/s — a host built to do real work, not just feed.

The memory the host carries is the third axis of composition, and it tracks the host-attach fork tightly. The coherent superchips use LPDDR5X — the low-power DRAM from the mobile world — soldered on Grace, but in detachable SOCAMM modules on Vera. The reasons are bandwidth-per-watt and density: NVIDIA’s GB200 profile pairs Grace with up to 480 GB of LPDDR5X at up to 512 GB/s; the host power setting belongs to the selected OEM profile, and NVIDIA's framing is roughly 2x the bandwidth at half the power of conventional socketed server memory. Vera pushes this to up to 1.5 TB at ~1.2 TB/s. In a power-bound rack where every watt spent on host memory is a watt not spent on accelerators, LPDDR5X's efficiency is the whole point. The price you pay depends on the generation, and the two are not the same bargain. Grace's LPDDR5X is soldered: capacity is fixed at manufacture and a memory fault is a package fault, with no module to pull and replace. Vera changes that — NVIDIA packages its LPDDR5X in SOCAMM modules specified as detachable, field-replaceable and capacity-upgradable, which is the point of the design. Serviceability and error protection are separate axes in any case: Grace documents ECC and channel sparing, so compare the RAS features and the repair unit each SKU actually supports rather than assuming LPDDR means no server-grade protection.

The discrete x86 hosts use socketed DDR5, and increasingly MRDIMM (multiplexed-rank DIMMs) on the Intel side to claw back the bandwidth gap — Granite Rapids reaches MRDIMM-8800 against AMD's DDR5-6000/6400 on EPYC Turin. Socketed memory offers composability (compare Vera’s detachable SOCAMM on its own terms): you size capacity to the workload (multiple terabytes for embedding tables, large KV-cache, or in-memory retrieval indices), you get full server-grade RAS, and you field-service or upgrade memory without touching the accelerators. The cost is power and density — registered DDR5 burns more per gigabyte than LPDDR5X, and the DIMM slots consume board area and a slice of the rack's thermal and electrical budget. The decision mirrors the host-attach fork: take a Grace superchip and you take its soldered LPDDR5X bargain; Vera’s SOCAMM changes the repair unit; take the discrete host and you can spend power to buy capacity, bandwidth, RAS, and serviceability on your own terms.

Composition is not a single answer. A real AI operation runs more than one workload shape, and the host-attach model, the GPU:CPU ratio, and the host memory that suit a pre-training trainer may differ from those of an agentic-inference fleet. So system composition is plural: compose distinct node and rack SKUs for the workloads that actually dominate your draw.

A training SKU may accept the coherent superchip ratio as shipped (2:1 on the GB200/GB300 NVL72 generation), profiled host application compute, host memory sized to the data pipeline not to the model, and every spare watt routed to accelerators. An agentic-inference SKU may need a larger host pool — many cores for orchestration and tool execution, large host memory for KV-cache and retrieval state, and a GPU:CPU ratio or separate CPU pool sized from CPU-seconds, memory and queueing evidence. A batch-inference SKU sits between them, throughput-bound and tolerant of a thinner host. Forcing one composition across mismatched profiles is how you end up simultaneously paying for unused host memory on the trainers and starving the agentic fleet of the CPU it needs. This is the same plurality the archetype framework demands at facility scale (Chapter 1.1) — here it lands inside the node.

CPU demand = 100 requests/s × 0.20 CPU-s/request = 20 busy cores. Provision ceil(20/0.70) = 29 cores, so the 32-core pool passes and 16 cores fail. Memory = 30 × 2.0 GB + 40 GB = 100 GB, within 128 GB. The 16-core service ceiling is 16 × 0.70/0.20 = 56 requests/s; the 32-core ceiling is 32 × 0.70/0.20, about 110 requests/s. At 120 requests/s, ceil(120 × 0.20/0.70) = 35 cores: the pool must grow or the CPU work must fall.

Add the CPU pool in the assumed base case, then trace queues, scheduler locality and request latency before adding GPUs. At or below 56 requests/s the thin host clears this CPU screen and the added pool may buy no service benefit. Grace’s fixed memory and Vera’s replaceable SOCAMM have different repair consequences; choose their supported FRUs independently of the CPU arithmetic.

Method: NVIDIA CUDA best-practices profiling guidance. Chapter 7.11 owns the next handoff.

Underneath all of it sits one more decision: how much of system composition do you let the vendor make for you. The rack-scale coherent platforms (GB200/GB300 NVL72 today, Vera Rubin next) ship as integrated units with the CPU:GPU ratio, the host CPU, the host memory, and the interconnect all fixed at the factory. You buy a rack, not a node, and the composition is a given. That is genuinely valuable — the integration risk of welding 72 GPUs, 36 CPUs, NVSwitch trays, and a liquid-cooling loop into one coherent domain is enormous, and the vendor has absorbed it. But it means the GPU:CPU ratio, the host ISA, and the host memory are not yours to tune. If your agentic workload wants a fatter host than the package ships, the lever inside the rack is a different SKU, not a different host. The lever outside it is to stop treating the accelerator-attached host as the whole CPU tier: tool execution, RL environments, preprocessing and orchestration can run on a network-connected CPU pool sized independently — NVIDIA positions Vera as standalone CPU infrastructure as well as a GPU host. A fixed CPU:GPU ratio inside a superchip does not fix the CPU:GPU ratio of the service; what decides is latency, data locality, isolation and transfer volume.

The discrete path keeps composition in your hands. You build the node — choosing AMD or Intel, the core count, the socketed memory, the GPU count per host, the ratio — and you own both the flexibility and the integration risk. For a fleet with heterogeneous workloads and a procurement strategy that refuses to single-source, this is the path that lets composition track the workload. The 2026 reality is that most large operators run both: bundled coherent racks where the workload genuinely needs the coherent domain (training, memory-hungry inference), and composed discrete nodes where composability and dual-source supply matter more. The decision is not which path is correct in the abstract — it is which workloads belong on which path, and that lands you right back at the fleet-plurality discipline above. → procurement and fleet composition in Chapter 7.11.