Scale-Out Topology, Sizing & Oversubscription

By the time you reach this chapter you have already chosen a protocol and transport (Chapter 8.4) and the switch, NIC, and DPU silicon (Chapter 8.3). What remains turns those components into an actual machine: how do you wire N GPUs together, and how much bandwidth do you refuse to pay for? Those are two views of one decision. Topology fixes the shape — how many tiers, how many switches, how long the longest path, how much bisection bandwidth the cluster sustains when every GPU talks to every other at once. Oversubscription fixes the price — how much of that theoretical bisection you actually build versus deliberately starve, because the workload will never use it.

A topology choice propagates deterministically into a switch count, an optics count, a cable-length distribution, a power draw, and an effective bisection bandwidth that caps the collective throughput of the parallelism plan running on top of it. What follows builds the topology taxonomy, derives the sizing math in scalable units, works the oversubscription fork that separates a training fabric from an inference one, and maps the parallelism strategy onto the wires. The back-end fabric is the second-largest line item after the accelerators, and a mis-sized decision here is paid quietly in lost goodput on every step of every job.

A handful of fabric families account for essentially every AI cluster shipping in 2026. They trade the same three currencies against each other: bisection bandwidth (how much the fabric can move when the traffic is all-to-all), switch and optic count (the cost and the failure surface), and worst-case hop diameter (latency and the blast radius of congestion). No family wins on all three; the right one is a function of scale and of which collective dominates your workload.

Fat-tree / Clos is the default and the safe choice. A folded multi-tier Clos with the right radix delivers full bisection bandwidth at every tier — genuinely non-blocking — and scales by adding tiers: a 64-port switch (32 down, 32 up) builds a three-tier fat-tree to 65,536 endpoints (k³/4); NVIDIA's 144-port Quantum-X800 reaches 10,368 NICs in just two tiers, and tens of thousands of GPUs in three. The cost is switch count and optics: a non-blocking three-tier Clos is the most switch- and transceiver-heavy topology there is, and at 100k-GPU scale the optics alone become a top-tier line item and the leading hardware failure source.

Rail-optimized fat-tree is the AI-specific refinement, and it is the dominant production pattern for NVIDIA-class clusters. Instead of one flat Clos, the scale-out fabric is split into rails — typically 8, one per GPU NIC in the server — where GPU i in every node connects to leaf switch i. Same-rail traffic (the common case for rail-aligned collectives) traverses a single leaf hop; cross-rail traffic falls back to NVLink inside the scale-up domain. The payoff is that the all-reduce, which is rail-aligned by construction, mostly stays one hop away, dramatically cutting spine load. Rail-only pushes this further: it removes the spine tier entirely, keeping only the rail leaves. Cross-rail traffic does not disappear — it detours through the scale-up domain, the move NCCL's PXN path already makes on rail-optimized fabrics (hop the NVLink domain to a GPU on the destination's rail, then cross without changing rails — NVIDIA, 2022). The MIT CSAIL/Meta proposal that named the pattern cuts network cost 38–77% versus a full-bisection fat-tree (arXiv, 2023–24); the price is scope — it assumes the collective profile stays rail-aligned, so all-to-all-heavy or failure-rerouted traffic pays the scale-up forwarding overhead, and the design is a bet on workload stability, not a free lunch.

Dragonfly trades bisection for switch count. Groups of switches are fully connected internally and sparsely connected to each other, yielding at most three hops endpoint-to-endpoint with far fewer switches and optics than a fat-tree of equal size — but lower bisection bandwidth, which is why it favors HPC and latency-bound patterns over bandwidth-bound all-reduce. OCS and torus is the hyperscaler-custom lane: Google's TPU pods use optical circuit switches to reconfigure the physical topology on demand (a 3D torus or twisted variant), decoupling logical from physical wiring and cutting worst-case diameter — Google's reconfigurable approach reduces a 1,024-chip worst case from 16 hops on a plain 3D torus to ~7. OCS removes a whole tier of packet switches and their optics, but it is a vertically-integrated play most operators cannot buy off the shelf.

Two more families bracket the taxonomy from opposite ends. Single-tier is the degenerate case worth naming because it is the simplest correct fabric wherever it fits: the entire back-end is one switch — one modular chassis, or one fixed box per plane — and most of the congestion machinery of Chapter 8.6 becomes moot, because there are no fabric links to collide on and no multi-path decisions to make. The ceiling is chassis radix: 2026-generation modular AI spines reach 576×800G or 1,152×400G in one box (~460 Tb/s), so up to 576 accelerators at 800G — or 1,152 at 400G — connect directly to one chassis and skip the leaf-spine tier entirely, along with its optics and cabling (vendor datasheets, 2026). Scheduled fabrics (Meta's DSF; the productized DES/DDC class) extend single-switch semantics beyond one chassis: leaf and fabric elements built on deep-buffer VOQ silicon behave as one logical switch — ingress traffic is sprayed as cells across all fabric links, and egress credits are granted before a packet is admitted — so congestion is prevented by scheduling rather than signaled and reacted to, and the ECMP/DCQCN tuning burden of Chapter 8.6 largely disappears. Meta reports its DSF at 18K×800G-GPU scale (Meta engineering, 2025), and productized scheduled fabrics list >27,000 800G ports in a single logical switch. The trade: the fabric's internals are single-vendor by construction, per-port cost is higher, and the scheduled domain is one administrative blast radius — the lock-in-for-determinism bargain the deep-buffer row of Chapter 8.3 prices.

The table reads left to right as choice into consequence, the same cascade as the workload archetype in Chapter 1.1. A non-blocking fat-tree buys you the highest optics count and the largest failure surface in the building. Dragonfly accepts lower bisection for far fewer transceivers — a rational trade only if your dominant collective is not bandwidth-bound. Because the three currencies move together, sizing the topology is where the cost is set.

A plane is a complete, independent copy of the scale-out fabric, and every accelerator connects to all of them. Split a NIC's bandwidth across its supported port configurations — an 800G NIC presenting as 2×400G, 4×200G, or 8×100G — and give each port to a different plane: N parallel leaf-spine fabrics, deliberately not interconnected, each carrying 1/N of every NIC's bandwidth, with the NIC or host stack balancing load across planes and containing per-plane failures. This is not a research topology; it is the shipping default. The B300-generation reference architecture attaches each GPU as 2×400GbE into two independent planes — on a NIC generation whose Ethernet ports top out at 400GbE, dual-plane is simply how 800 Gb/s attaches (→ Chapter 8.3) — with NCCL or the NIC's onboard plane load balancer spreading traffic, and a failed plane degrading the job to half bandwidth instead of killing it (NVIDIA reference architectures, 2025–26). The MRC/UET NIC generation extends the same move to four and eight planes with per-packet, hardware plane balancing (→ Chapter 8.4).

Keep the vocabulary straight, because the industry does not: a rail is a same-local-rank affinity group within one fabric — all the rank-i NIC ports across servers, commonly mapped to one leaf (though a big leaf can carry several rails) so same-rank traffic stays one hop away; a plane is which independent fabric a port belongs to. The two compose: each plane of a multi-planar cluster is itself rail-optimized, exactly as a single-fabric cluster would be.

The reason planes matter is arithmetic. In a symmetric non-blocking two-tier Clos — half of each leaf's ports facing endpoints, half facing spines — endpoint count scales with the square of switch radix: 64 ports of matching speed reach 2,048 endpoints; 512 reach 131,072. But radix at full port speed is capped by the switch ASIC: a 51.2 Tb/s chip is 64×800G or 512×100G, port count trading against port speed. Planes convert that trade into scale. Run each NIC at 1/N speed per plane and the per-plane radix multiplies: the same silicon that connects 2,048 endpoints as one 800G fabric connects 131,072 GPUs as eight 100G planes — the design published in the OpenAI-led MRC/SRv6 paper alongside its four-plane production configurations, and the same two-tier-to-~128K arithmetic NVIDIA's multiplane analysis reaches (OpenAI et al., 'Resilient AI Supercomputer Networking using MRC and SRv6,' 2026; NVIDIA, 'High-speed Networking for Giga-Scale AI Factories,' 2026). Against a full-bisection three-tier build at that scale, the MRC/SRv6 paper counts roughly two-thirds the optics and three-fifths the switches, with a worst-case path of three switch hops instead of five to seven — which is why the frontier-scale consensus is hardening into two tiers, more planes, bigger spines, not a third tier. The price: per-plane speed drops (fine for spray-capable transports, a real constraint for single-path flows), the NIC must balance and fail over across planes, and the symmetry discipline of a fat-tree now applies N times over. → the congestion machinery a plane no longer shares in Chapter 8.6; the transports that made multi-plane native in Chapter 8.4.

The professional way to size a fabric is not to count GPUs — it is to count scalable units (SUs). An SU is the smallest repeatable block of GPUs, leaf switches, optics, and cabling that you replicate to grow the cluster. Sizing in SUs is what keeps the BOM, the cabling plan, the fault domains, and the procurement schedule tractable as a cluster ramps from one SU to dozens. In NVIDIA's GB200 SuperPOD reference, 1 SU = 8 rack-scale systems = 576 GPUs; a 16-SU cluster of 9,216 GPUs needs a switch BOM on the order of a thousand boxes across leaf, spine, and core for a non-blocking three-tier compute fabric, plus a separate storage fabric and a separate management fabric.

The math that turns an SU into a BOM is mechanical once the radix and the blocking factor are fixed. For a non-blocking fat-tree built from radix-k switches, each tier must provide as much uplink bandwidth as downlink bandwidth — so for every port facing the GPUs there is a port facing the next tier up. That 1:1 uplink:downlink rule is the entire definition of 'non-blocking,' and it is also the entire cost of it: half of every switch's ports, and the optics on them, exist solely to preserve bisection bandwidth. Relax the rule to 2:1 (two downlinks per uplink) and you delete a large fraction of the spine switches and their transceivers — which is precisely the oversubscription lever the next section turns.

Three counting traps recur. First, the optics dominate the back-end BOM, not the switches — at 100k-GPU scale a cluster carries tens of thousands of miles of fiber and the transceiver line can rival the switch line. Second, the storage and management fabrics are not free: the compute fabric is the headline, but a production SU also carries a non-blocking storage fabric (e.g. 16x800G per SU) and an out-of-band management network (Chapter 8.7). Third, the reach budget, not the radix, often caps the physical topology: passive DAC dies around 1–2 m at 800G, active copper (AEC) stretches to 3–7 m, and beyond that you are paying for optics on every link — so the cable-length distribution implied by your floor plan is part of the sizing math, not an afterthought.

Fabrics are bought once and grown many times, and the two motions are asymmetric: growth lives in the leaves; regret lives in the spine. Adding a scalable unit is additive — new racks, new leaf switches, new cables landing on spine ports that were waiting for them. Widening a spine after the fact is not: every leaf's uplinks must be re-spread across the new spine set to preserve the symmetry the collectives depend on, which means touching cabling fabric-wide on a live cluster. NVIDIA's reference architecture says the quiet part: even when a scalable unit deploys partially, the fabric should be designed and cabled for the full SU, with the unused portion left dark (NVIDIA B300 RA, 2025). The professional pattern is to fix the target scale first, size — and at minimum cable — the spine for it, and let utilization, not architecture, grow with the fleet.

Two sizing levers fall out. Interconnect speed versus spine width: at a fixed leaf uplink budget, running leaf–spine links at half the port speed doubles the spine switch count, each half-populated at initial scale — more boxes today, but the headroom to double the cluster by filling ports; running them at full speed halves the spine and arrives fully populated — cheaper now, but the next growth step is a redesign, and each spine failure removes twice the bandwidth per leaf. The uplink speed floor (an engineering implication, not a vendor rule): keep fabric links at least as fast as the host attach rate. A single unsplit NIC-rate flow cannot fit inside a slower fabric link, so it congests structurally at that bottleneck — no flow-level load balancer rescues a 400G flow on a 200G path; per-packet spray dissolves the single-flow constraint, but only for transports that spray and reorder (→ Chapter 8.4).

Deliberate overprovisioning — under-subscription — is a named tool, not an accident. Meta's production RoCE fabric, hit by ECMP flow collisions, doubled rail-switch uplink bandwidth to run 1:2 under-subscribed; once better load balancing landed, it trimmed the ratio to 1:1.125 rather than all the way to 1:1 — explicitly keeping buffer for up to two link failures (Meta, SIGCOMM 2024). That is the honest arithmetic of a fabric that must hold its blocking factor through real-world link loss: a few percent of extra uplink capacity, bought once, doubles as failure margin and ECMP smoothing. Rack granularity is the other quiet sizing input: rack-scale systems land NICs in multiples the switch radix may not divide. A GB200 NVL72 rack presents 18 NICs per rail (four GPUs per compute tray, 18 trays, one rail per tray position), so a 64-port leaf running 18 down and 18 up strands 28 ports — the mismatch that made 144-port (8×18) switch radixes the clean fit for NVL72-era fabrics (SemiAnalysis, 2024). Count ports against the rack unit and the rail structure before trusting any aggregate-bandwidth arithmetic; the BOM error hides in the remainders.

The fork is set by coupling. Oversubscription is the ratio of downlink bandwidth to uplink bandwidth at a tier — a 1:1 fabric provisions as much bandwidth leaving a tier as entering it (non-blocking); a 3:1 fabric provisions a third, betting the offered traffic never simultaneously demands full bisection. Whether that bet is safe or catastrophic depends entirely on the workload's coupling.

Synchronous training must be 1:1 non-blocking in the GPU east-west path. A pre-training step is dominated by all-reduce/all-gather collectives that move gradient and activation tensors across the entire data-parallel group on every iteration. Those collectives are bandwidth-bound and bursty in lockstep — every GPU hits the fabric at the same instant — so any oversubscription throttles the collective, stalls the slowest rank, and collapses model FLOPs utilization (MFU). There is no statistical multiplexing to exploit: the traffic is synchronized by construction, not statistically independent. Oversubscribe a training fabric and you do not save money — you buy a slower supercomputer that costs the same to run and finishes later.

Inference tolerates 2:1–3:1, and refusing to oversubscribe it wastes capital. An inference request fits inside a node or a small scale-up domain (Chapter 8.2), so most traffic never touches the scale-out spine; what does is statistically independent across requests, so a tier can multiplex many bursty flows and rarely saturate. A 2:1 'optimized' fabric cuts back-end cost roughly 31% versus non-blocking (a contested, largely single-source figure), and that saving is real money you redeploy into geo-distribution, uptime, or more accelerators. Build a non-blocking fabric for an inference business and you strand a third of your back-end budget on bisection bandwidth that never carries a packet — the anti-pattern Chapter 1.1 names.

Real clusters are tiered, so the ratio varies by tier. The bottom tier (leaf-to-GPU) stays non-blocking even on inference fabrics — the leaf is cheap and HOL blocking there is unforgiving. Oversubscription is applied up the tree, at the inter-zone and inter-building tiers where flows have aggregated and statistical multiplexing is strongest. Meta has publicly run as high as 7:1 on parts of a 24k-H100 fabric, and on training fabrics has run RTSW uplinks deliberately under-subscribed (1:2) as a congestion mitigation — the ratio is a per-tier knob, not a single global number.

A non-blocking fabric is necessary but not sufficient — you also have to place the parallelism on it correctly, or you starve the collective the fabric was built to serve. Match each parallelism dimension's bandwidth appetite to the tier that can feed it, working outward from the fattest pipe.

Tensor parallelism (TP) and expert parallelism (EP) belong inside the scale-up domain. These are the most bandwidth-hungry dimensions — TP shards a single layer across GPUs and exchanges activations every layer; EP routes tokens to experts with all-to-all on every MoE layer. They demand the ~1.8 TB/s/GPU of NVLink — a bidirectional aggregate, ~900 GB/s each way (NVL72 = 130 TB/s rack aggregate) — roughly 18x the ~400G scale-out NIC's ~50 GB/s per direction. So fit TP and EP entirely inside the scale-up domain; the moment they spill onto the scale-out fabric, the collective runs at NIC bandwidth instead of NVLink bandwidth and throughput falls off a cliff. This is the deepest reason scale-up domain size is a workload decision — a 72-GPU NVL72 domain admits far wider EP (and thus larger MoE models served efficiently) than an 8-GPU HGX domain. → scale-up fabric in Chapter 8.2; wide-EP economics in Chapter 10.11.

Pipeline parallelism (PP) and data parallelism (DP) ride the scale-out fabric. PP passes activations between pipeline stages — point-to-point, latency-sensitive but not bandwidth-crushing — and DP runs the gradient all-reduce across replicas. Both tolerate the scale-out fabric by design: PP overlaps communication with computation, and DP's all-reduce, while bandwidth-heavy, is the collective the rail-optimized fat-tree is laid out to serve one hop at a time. Place DP groups along rails so the all-reduce stays on-rail, and the spine carries only what genuinely must cross rails.

Get the mapping wrong and the damage is invisible in a topology diagram and brutal in production: a perfectly non-blocking fabric, fully provisioned, running a job whose TP dimension landed across the scale-out path — every layer's activation exchange crawling at NIC speed, MFU on the floor, and a network team insisting the fabric is healthy, because it is. Topology-aware scheduling (Chapter 10.2) exists to prevent exactly this; the fabric and the scheduler are co-designed, not independent.

Every topology decision resolves to a point in a three-axis space — cost, reach, and blocking — and the three are coupled, so you cannot optimize one without paying in the others. Blocking (the inverse of bisection bandwidth you provision) is set by the oversubscription ratio: non-blocking is the most expensive and the only acceptable choice for synchronous training. Reach is set by physics: at 800G and 1.6T the copper cliff arrives within a couple of meters, so a floor plan that spreads racks out converts copper links into optical links and multiplies the transceiver count — meaning the building layout is a fabric-cost variable. Cost is the resultant: it rises with switch tiers (more radix to cross), with non-blocking provisioning (more spine), and with optical reach (more transceivers).

The practical optimization is to spend copper where you can and optics where you must, and to push as much traffic as possible into the cheapest tier. Rail-optimized topologies are popular precisely because they keep the dominant collective one leaf hop away — minimizing the spine traffic that forces expensive optical links. Compact, dense floor plans keep more links inside the copper reach budget. And the SU boundary is drawn, in part, to keep intra-SU links on copper and reserve optics for SU-to-SU. The fabric that looks cheapest on a per-port basis is rarely cheapest in the building; the all-in number is dominated by how many of your links crossed the copper cliff into optics.

The recurring fabric-sizing mistakes all share a root cause: sizing from a component spec or a topology preference instead of from the workload's coupling and parallelism plan. Five are worth naming:

• Non-blocking fabric for an inference business. Building 1:1 bisection for a workload whose requests fit inside a node strands ~31% of the back-end budget on bandwidth that never carries a packet. Oversubscribe the upper tiers and spend the saving on uptime and proximity.
• Oversubscribing a training fabric. The symmetric error. A 2:1 ratio on a synchronous all-reduce throttles every step forever — you bought a permanently slower supercomputer to save capex that lost MFU repays many times over.
• Parallelism spilling out of the scale-up domain. Placing TP or wide EP across the ~400G scale-out path (~50 GB/s per direction) instead of the ~1.8 TB/s NVLink domain (~900 GB/s per direction) runs the most bandwidth-hungry collective ~18x too slow per direction on a fabric that reports healthy. Fit TP/EP inside the scale-up domain or accept the collapse.
• Growing the spine reactively. Sizing the spine for this quarter's GPU count and widening it per expansion breaks leaf–spine symmetry and forces fabric-wide recabling on a live cluster. Fix the target scale; build — or at least cable — the spine for it on day one, and grow in leaves and SUs. When the target outruns one fabric's radix, scale with bigger-radix spines or added planes before conceding a third tier.
• Sizing in GPUs instead of SUs. Ad-hoc GPU counts produce ad-hoc cabling, irregular fault domains, and a BOM that does not replicate. Draw the SU boundary first; size, cable, and procure in SUs.