Network Fundamentals & AI Traffic Characterization

In a single-GPU world the network is an afterthought — feed data in, ship results out. In a frontier AI cluster the interconnect is a first-class component of the computer, and for the largest synchronous jobs it gates all the others. A modern training run is one program spread across tens of thousands of accelerators that must agree, repeatedly and synchronously, on the value of a shared set of gradients. That agreement is a collective communication — an all-reduce, all-gather, or reduce-scatter — and until it completes, every GPU that finished its share of the work sits idle at a barrier waiting for the slowest participant. So the network does not carry bytes between the computations; it sequences the computation itself. Misjudge it and the expensive accelerators run at a fraction of their FLOPS, burning wall-clock while they wait on each other.

The rest of Part 8 builds on the primitives set here: the three-network model (scale-up, scale-out, scale-across) and why its internal boundaries are moving; the collective primitives and how parallelism strategies map onto each network tier; the tyranny of the tail and the BSP barrier semantics that turn a one-in-a-thousand slow link into a fleet-wide stall; the divergence between raw link rate and delivered goodput that decides whether a fabric earns its capital; the two traffic profiles — training versus inference — and job-completion-time as the north-star metric that reframes every downstream networking decision.

An AI cluster is not one network. It is a hierarchy of at least three, each with a different bandwidth, latency, reach, and cost-per-bit, and the single most consequential network-design decision is where you draw the boundaries between them. The hierarchy spans roughly three orders of magnitude in per-GPU bandwidth and four in latency, and traffic that lands on the wrong tier is either ruinously expensive or ruinously slow.

Scale-up is the memory-semantic fabric inside a tightly-coupled domain — historically a node, now including the 72-GPU GB200 NVL72 rack and NVIDIA’s March 2026 forecast of a 576-GPU Rubin Ultra domain across eight racks, each with its own topology and software contract. It is the fastest and most expensive tier: NVLink 5 delivers ~1.8 TB/s per GPU bidirectional (~0.9 TB/s per direction), an NVL72 rack carries ~130 TB/s of aggregate injection bandwidth, and supported peer load/store operations let a GPU read or write another GPU’s memory without a host-staged message; the platform’s ordering rules still apply, and remote access does not imply local latency or cache coherence. This is where you want tensor-parallel and expert-parallel traffic to live, because those collectives are bandwidth-hungry on every layer. → Chapter 8.2.

Scale-out is the packet-switched fabric that stitches domains into a cluster of tens of thousands of GPUs. It is roughly an order of magnitude slower per GPU — a 400 Gb/s-per-direction NIC (50 GB/s) is ~1/18th of NVLink-5's ~0.9 TB/s-per-direction figure; 800 Gb/s per direction (100 GB/s) makes the ratio ~9:1 before overhead — and an order of magnitude cheaper. The semantics are message-passing (RDMA over InfiniBand or RoCE/Ethernet), and this is the natural home for data-parallel gradient all-reduce and pipeline-parallel point-to-point. → Chapter 8.4 and Chapter 8.5.

Scale-across is the newest tier and a creature of the power-bound era: the WAN/DCI fabric that couples multiple campuses when no single site can be energized at the scale a frontier run demands. Crossing a site boundary does not select the algorithm; measured RTT, bandwidth, and the job's communication pattern do. Qualified cross-campus or intra-metro paths can preserve synchronous methods, while longer regional paths move the design toward hierarchical or relaxed-synchrony methods and compression when the bandwidth budget demands it. Power scarcity can push a frontier lab toward multiple campuses when one site cannot supply the required compute, but the protected-route budget must still admit each workload; the grid constraint does not make synchronous traffic fit a WAN. → Chapter 8.8; the power-bound rationale in Chapter 16.1.

Around these three sits a fourth network that every deployment has and no fabric budget should forget: the front-end network — the CPU-side Ethernet carrying data ingestion, checkpoint traffic, storage access, user/API serving, and cluster services. Meta describes its training clusters as running two independent networks — a front-end for ingestion, checkpointing, and logging; a back-end for training — and NVIDIA's reference architectures split the same functions into distinct storage and in-band management fabrics beside the compute fabric (Meta, SIGCOMM 2024; NVIDIA reference architectures, 2025). It is ordinary Ethernet in kind but not in size, and its sizing is a live decision with an order of magnitude inside it. The front-end rate follows concurrent ingestion, checkpoint save/restore, KV movement and service traffic, together with their deadlines. Count the NIC ports actually attached to storage and the uplinks shared with management; dividing a tray aggregate by GPU count does not establish workload demand. A reference configuration is useful only with its host connections, storage-device connections and oversubscription boundary intact. The NVIDIA GB200 reference architecture describes those as a separate fabric. Front-end sizing lives with Chapter 9.2 and Chapter 9.3, and management recovery with Chapter 8.7.

Distributed training is built from a small vocabulary of collective communication primitives, and almost all the traffic in a cluster is one of them. All-reduce (sum a tensor across all ranks, return the result to all) is the workhorse of data-parallel gradient synchronization. All-gather and reduce-scatter (the two halves of which a ring all-reduce is composed) move sharded weights and activations under tensor and sharded-data parallelism. All-to-all is the signature of mixture-of-experts: each token is routed to its selected experts; the byte matrix can be sparse and skewed, stressing a destination or a cut rather than loading every path uniformly. Point-to-point sends move activations forward and gradients backward across a pipeline stage.

The engineering content is the mapping: which parallelism dimension generates which collective, and which tier can carry its exposed exchanges within budget. Get the mapping right and each collective lands on a fabric sized for it; get it wrong and you push a bandwidth-bound collective onto a tier that cannot feed it, and the job stalls. The four parallelism dimensions are nested precisely so they can be mapped onto the bandwidth hierarchy.

The table is a placement hypothesis to test. The design move is to keep the leftmost, highest-frequency rows (TP, EP) inside the scale-up domain where the local NVLink fabric’s bandwidth can keep the collective off the critical path when the selected schedule overlaps its completion, and to let the lower-frequency rows (PP, DP) spill onto the cheaper scale-out fabric when their measured timing and overlap tolerate lower bandwidth or oversubscription. This is why domain size dominates the fabric design: each GPU added to the domain can keep another chunk of TP/EP traffic off the slower fabric when the runtime places that state and exchange locally. It is also why mixture-of-experts inference reshaped fabric design — wide expert parallelism benefits when its exposed dispatch/combine traffic stays on NVLink, but enough expert capacity and a tested hierarchical NVLink/RDMA schedule can justify crossing domains. → MoE inference and EP degree in Chapter 8.2.

One fact separates AI networking from every other kind of data-center networking: a synchronous training step is a bulk-synchronous-parallel (BSP) computation, and a BSP barrier runs at the speed of its slowest participant. When ten thousand GPUs all-reduce a gradient, the collective is not done when the median link finishes — it is done when the last one does. Average bandwidth is almost irrelevant; what governs wall-clock is the tail. A fabric with superb median latency and a fat 99.9th-percentile tail will deliver worse training throughput than a fabric with mediocre median latency and a tight tail. This is the tyranny of the tail, and it inverts the intuition every web-scale network engineer brings to the problem.

The traffic pattern depends on the collective algorithm: parameter-server aggregation and some tree/reduction fan-ins produce synchronized many-to-one incast, while ring all-reduce produces synchronized neighbor flows with a long dependency chain, so its slowest transfer stretches the collective tail. The selected algorithm's traffic matrix determines whether switch buffers, flow control (PFC), or packet loss drive that tail. One slow link, one congested switch, one mis-tuned congestion-control loop, and the barrier that should have closed in microseconds drags for milliseconds, multiplied across every step of a months-long run. Circulating industry write-ups put percentages on this effect, but the concrete failure mechanism is a training barrier stretched by network delay: in a 175B-parameter job, a congested queue or a retransmission can leave ready GPUs waiting for the last rank. Those percentages cannot size a fabric without the model, GPU count, collective algorithm, loss process, runtime and step-time denominator that produced them; a theoretical packet-loss result is a sensitivity to test, not a measured loss of compute on the buyer's cluster. The job's own completion trace decides whether removing that delay recovers enough accelerator time to justify the fabric change. What transfers between clusters is the mechanism: barrier-coupled work turns a fabric defect into fleet-wide idle.

A 400 Gb/s port does not deliver 400 Gb/s of useful gradient synchronization: protocol overhead, congestion, retries and barrier wait stand between the rate on the invoice and the training run’s wall-clock. A faster port earns its price when useful communication advances the job sooner. MAC link rate is the Ethernet service rate, distinct from coded SerDes signaling. Payload throughput is useful delivered bytes divided by transfer time in a named direction; payload efficiency divides that rate by the corresponding MAC byte rate. Collective algorithm bandwidth is the stated message size divided by operation time; NCCL bus-bandwidth normalization depends on operation and ranks and is not switch telemetry. MFU divides useful model FLOPs by elapsed seconds times participating accelerators’ declared peak FLOPs/s at the selected precision. Exposed communication is dependency delay after useful overlap, divided by step time when reported as a fraction. Job goodput counts useful completed work per wall-clock time across the declared checkpoint/recovery window; that operations ledger lives in Chapter 14.1.

The protocol choice is the cleanest illustration. InfiniBand's effective throughput and latency belong to the tested adapter, switch, collective and message size; a switch-hop latency cannot stand in for application completion. Untuned RoCEv2 over Ethernet can show materially higher and more variable latency with a long tail when congestion or reordering triggers the selected NIC's retry behavior; go-back-N receivers repeat traffic that a receiver with supported selective retry or out-of-order placement can avoid. NVIDIA's 2024-10-28 Colossus announcement reports roughly 95% data throughput for Spectrum-X while training Grok. That is one vendor-reported workload and configuration, not a universal Spectrum-X result or evidence for Ultra Ethernet deployments. Meta's 2024-08-05 production account describes the topology, routing and congestion tuning that made RoCE viable for its distributed training. The lesson is that tuning and topology earn their cost when they shorten the same job's exposed communication; a faster link earns its cost when that job is still bandwidth-limited. The NCCL-tests definitions keep collective comparisons on the same denominator. → transport qualification in Chapter 8.4.

For n=64, each rank sends W=2(n−1)S/n=2×63/64×4 GiB=7.875 GiB=8,455,716,864 bytes. Healthy communication is W/(45×10⁹ bytes/s); step time is 800 ms + max(0, W/(45×10⁹)×1,000−150 ms), or 840 ms at 10 ms precision: pass. One lost uplink leaves R=45×31/32 GB/s; the same expression gives 840 ms at 10 ms precision: pass. Unrounded times differ although displayed results coincide. Keep TP local and admit DP only after the built fabric sustains the assumed tail service; this step alone does not justify a faster NIC.

The flip is growth on a fixed surviving cut. With n/2 senders, R(n)=31×45×2/n GB/s and communication is 2(n−1)×4×2³⁰/(2×31×45×10⁹) seconds. It must be no more than (1,100−800+150) ms. The crossover is about 147 ranks: the largest balanced even group that passes is 146; 148 fails. Extra ranks need extra leaf ports: this sensitivity does not enlarge the original two leaves. Fewer bytes or more overlap moves the crossover outward. The NCCL-tests method supplies normalization; Chapter 8.5 tests spine loss and counts expansion.

Training and inference often produce different traffic, but the label does not select the fabric. Synchronous collectives are bursty and bisection-hungry, so measured step-time and MFU targets often justify high bisection. Many local inference requests permit upper-tier multiplexing, while distributed MoE, KV movement, and prefill/decode disaggregation can require substantial scale-out capacity. Published fabrics span 1:1, 2:1–3:1, and reported 7:1 examples. Chapter 8.5 turns those observations into a per-tier ratio through its locality, exposure and failure-headroom tests; this chapter supplies the traffic characterization those tests read.

The 2025–2026 twist is that inference traffic is no longer trivial. Reasoning models emit long decode sequences, mixture-of-experts routing creates real all-to-all traffic, and prefill/decode disaggregation moves the KV-cache across the fabric — so inference, once a single-node afterthought, now exercises the scale-up domain hard. The characterization still differs from training (request-SLO-bound, with synchronization inside distributed kernels), but the gap is narrowing, and a fabric scoped for last-generation single-node inference can be undersized for MoE reasoning serving. → inference fabric demands in Chapter 1.3; MoE all-to-all in Chapter 8.2.

The transfer allowance is 250−180=70 ms. Eight requests carry 8×1 GiB=8,589,934,592 bytes. Across racks, completion is 180 ms +8,589,934,592/(45×10⁹)×1,000 ms, about 370 ms: fail. With local service of 300 GB/s it is about 210 ms: pass, rounded to 10 ms. Select local prefill/decode placement, accepting its memory and scheduling constraints to avoid the request-tail miss.

Batch size reverses the choice: B×2³⁰/(45×10⁹) ≤0.070 gives B≤2.93. Two concurrent transfers give about 230 ms: pass; three give about 250 ms but fail the unrounded limit. Select cross-rack placement at two transfers or fewer, or acquire more service capacity. Hand Chapter 9.7 the KV representation, bytes/request, burst, ownership-transfer event and time allowance; it owns KV sizing. The Dynamo disaggregated-serving design supplies the mechanism. Validate the joint request tail in Chapter 13.7.

Every networking decision in Part 8 ultimately answers to one metric, and it is not bandwidth, latency, or port count — it is job-completion-time (for training) and its inference twin, tokens-per-second-per-dollar at the SLO. JCT is the wall-clock from job start to a trained model, and it folds in everything the spec sheets hide: the tail, the congestion, the goodput, the failures and restarts. A fabric is good if and only if it lowers JCT per dollar of capital and per watt of power — and a fabric can have best-in-class link rate, best-in-class median latency, and still lose on JCT because its tail or its reliability is poor.

Reframing networking around JCT changes what you optimize. You stop chasing peak bisection bandwidth as a vanity number and start chasing the combination that minimizes wall-clock: tight tail, high goodput, fast recovery from the inevitable link and switch failures (which the Llama-3 run attributed ~8.4% of interruptions to). It also re-prioritizes the whole part — congestion control, in-network reduction, adaptive routing, and topology matter because they move JCT, not because they look impressive in a bake-off. This is the same shift from availability to goodput that the reliability chapter makes for the facility: the question is never 'is the link up?' but 'how much useful training did the fabric deliver this week?'. → goodput vs availability in Chapter 12.2; checkpoint/restart math behind failure recovery in Chapter 9.4.

Choose the tier boundary from the exchanges that remain exposed, then carry healthy and degraded deadlines into the fabric order. Keep a group local when crossing fails that budget; split it when memory capacity demands the crossing and the measured schedule can afford it. Choosing on port rate alone buys capacity the job may never use while leaving its slowest dependency unpaid for.