Topology-Aware & Rack-Scale Scheduling

Chapter 10.1 established the scheduling plane as the control loop that turns a fleet of accelerators into a service. This chapter is about the single hardest constraint that loop has to respect on 2026-era hardware: the network is not flat. Between any two GPUs in a modern AI cluster there are three radically different classes of link, and the bandwidth between them differs by roughly an order of magnitude at each step. Inside a node, GPUs talk over NVLink at terabytes per second. Inside a rack-scale NVLink domain — a GB200 NVL72 is 72 GPUs across 18 compute trays sharing 130 TB/s of aggregate NVLink — they still talk at scale-up speeds as if they were on one giant board. The moment a collective has to leave that domain, it drops onto the scale-out fabric: a ~400G or 800G NIC, an order of magnitude less per-GPU bandwidth, and a different failure and congestion regime. A scheduler that ignores this hierarchy will cheerfully place a tightly-coupled job with eight GPUs in one rack and eight in the next, and the job will run — at a fraction of its rightful throughput, forever, with no error logged.

The fork is topology-blind vs topology-aware placement, and the downstream cost of choosing blind is paid in goodput every step of every job for the life of the run. We walk the bandwidth cliffs and how you discover them; the two production mechanisms for defending them — Slurm block scheduling with IMEX, and Kubernetes ComputeDomains over Dynamic Resource Allocation; how to map a parallelism strategy onto the fabric so the right collectives stay local; and the fragmentation problem that makes the rack, not the node, the natural unit of allocation on rack-scale systems.

Hold three numbers in mind, because everything in this chapter is a consequence of the gaps between them. NVLink (scale-up, within the coherent domain): NVLink 5 on Blackwell delivers 1.8 TB/s bidirectional per GPU; an NVL72 rack aggregates to 130 TB/s. PCIe (host-local, GPU-to-CPU or GPU-to-NIC): Gen5 x16 is ~64 GB/s each way — the path KV-cache and host staging traverse, and a bottleneck the moment data must touch the CPU. Scale-out NIC (inter-node, over InfiniBand or RoCE Ethernet): a single ~400 Gb/s port is 50 GB/s; even an 8-rail node at 3,200 Gb/s is ~400 GB/s aggregate. The rule of thumb practitioners carry is that scale-up per-GPU bandwidth is roughly 5–18x scale-out, and the consequence is unambiguous: the most bandwidth-hungry collectives must be fit inside the scale-up domain, and the scheduler is the thing that decides whether they are.

Bandwidth does not degrade gracefully as a job spreads; it steps down discontinuously at each boundary. A tensor-parallel all-reduce that fits in one NVLink domain runs at NVLink speed; the same all-reduce split 9-and-9 across two domains runs at NIC speed for the cross-domain portion and stalls every other rank waiting on it. NVIDIA's own Slurm guidance states the case flatly: when a job crosses the NVLink-domain boundary, performance "drops sharply," which is why their block plugin treats the domain as a hard constraint rather than a best-effort preference. The failure mode is insidious because it is silent — the job completes, the loss curve descends, nothing alarms — but MFU sits 20–40% below where the hardware should deliver, and the bill for that gap compounds across the entire run.

A scheduler can only defend a topology it can see, and discovery is where most topology-aware deployments quietly fail. There are two postures, and they correspond to a trust decision. Trusted discovery reads the fabric from labels the platform asserts: NVIDIA exposes an NVLink clique identifier (the nvidia.com/gpu.clique node label in Kubernetes; equivalent block definitions in Slurm's topology.yaml) that tells the scheduler which nodes share a coherent NVLink partition. PCIe and NUMA affinity come from the driver and from the node's hardware topology. Scale-out topology — which leaf switch a node hangs off, which rail a NIC belongs to — comes from cabling databases, LLDP, or a topology file the operator maintains. Trusted discovery is fast and is how production schedulers run day-to-day.

Verified discovery distrusts the labels and measures. At node admission and after every repair you run point-to-point bandwidth probes (nvbandwidth, NCCL all-reduce/all-gather benchmarks, p2pBandwidthLatencyTest) and compare against the expected matrix for the asserted topology. This matters because the labels can be right while the physical link is degraded: a marginal NVLink lane that has fallen back to a lower width, a transceiver running hot, a cable seated badly. The label still says "one clique, full bandwidth"; the hardware delivers half. A scheduler trusting that label places a tightly-coupled job onto a cliff it believes isn't there. The discipline that separates a 96%-goodput fleet from a 90% one is verifying the topology the scheduler will trust — gating nodes into the schedulable pool only after they pass the bandwidth matrix, not merely after they boot. Verification belongs to the burn-in and health pipeline of Chapter 10.6; the scheduler consumes its verdict.

Slurm remains the dominant scheduler for large training fleets, and its answer to rack-scale topology is block scheduling (the topology/block plugin, hardened for NVL72 in 2025). The model is simple and powerful: you declare, in topology.yaml, that a set of nodes forms a block corresponding to one NVLink domain — for an NVL72 that is 18 compute nodes carrying 72 GPUs. The scheduler then treats the block as an atomic locality unit. A job requesting up to 18 nodes is consolidated into a single block, so its tensor- and expert-parallel collectives never leave NVLink. A job larger than one block spans whole blocks rather than being smeared arbitrarily, so the cross-domain traffic is confined to the parallelism dimensions that tolerate scale-out bandwidth.

Two refinements make this practical rather than rigid. The --segment argument lets a user declare the atomic group size their job needs — for example --segment=4 lets a 12-node job be split into three 4-node segments that the scheduler can pack flexibly across blocks while still keeping each segment's locality intact. This is the release valve for fragmentation: instead of demanding one contiguous 12-node hole, the scheduler can fill three smaller holes, lifting utilization without abandoning topology. The second refinement is the switch/nvidia_imex plugin (Slurm 24.05+), which manages IMEX channels — the driver-level memory-export/import access control that lets GPUs in a multi-node NVLink domain address each other's memory. Slurm provisions and tears down the IMEX channel per job, so two jobs sharing a physical NVLink domain cannot accidentally read each other's GPU memory. Block scheduling without IMEX management is a security and correctness gap on shared NVL72 hardware, not merely a tuning omission.

Kubernetes arrived at rack-scale topology from the opposite direction. Its original device-plugin model exposed a GPU as a flat, countable, location-free resource — nvidia.com/gpu: 8 — which is structurally blind to NVLink domains. The fix is Dynamic Resource Allocation (DRA), the resource API that lets a workload request a structured, parameterized resource rather than an opaque count, and on top of it NVIDIA's ComputeDomains abstraction. A ComputeDomain represents reachability between the distributed workers of a multi-node job: when its pods are scheduled, the platform dynamically creates the IMEX domain that lets those pods' GPUs address one another's memory over NVLink, and tears it down when the job ends. The IMEX channel surfaces inside each container as a device file, so a CUDA or NCCL application behaves as if all the GPUs sat on one board — the rack-scale illusion, delivered through the scheduler.

Two constraints define how you design around this. First, placement still needs an affinity rule: pods must be steered onto nodes that share an NVLink partition, expressed by matching the nvidia.com/gpu.clique node label, or the ComputeDomain spans a boundary it cannot bridge. The clique label is the Kubernetes analogue of the Slurm block. Second — and this is the sharp edge of the 2026 implementation — there is at most one ComputeDomain (one IMEX channel) per node, because only one IMEX daemon runs per node. If a ComputeDomain claims only part of a node's GPUs, the remaining GPUs on that node cannot join a different ComputeDomain and are effectively stranded. That single constraint pushes you hard toward whole-node, and in practice whole-rack, allocation for multi-node NVLink jobs: partial-node sharing across distinct multi-node domains is not expressible today. The platform stack to run this is specific — Kubernetes 1.32+ with the DRA APIs enabled, recent GPU Operator and driver — and getting the versions wrong yields silent fallback to flat scheduling, which is exactly the cliff you were trying to avoid.

Topology-aware scheduling is only half the bargain; the other half is the job declaring its parallelism so the scheduler can place the right dimension on the right tier. 3D (and 4D) parallelism uses four collective patterns with sharply different bandwidth appetites, and the entire art is matching appetite to tier. Tensor parallelism (TP) all-reduces activations on every layer — the most bandwidth-hungry pattern, and the one that must live inside the NVLink domain. Expert parallelism (EP) for MoE models does all-to-all token routing; wide EP (EP32 and beyond) is precisely what a 72-GPU NVLink domain unlocks, because the all-to-all stays on NVLink instead of hitting the scale-out fabric. Pipeline parallelism (PP) passes activations point-to-point between stages — low volume, latency-sensitive, tolerant of crossing the domain boundary. Data parallelism (DP) all-reduces gradients once per step — high volume but infrequent and overlappable, the canonical dimension to push onto scale-out.

The placement rule that falls out: TP and EP inside the scale-up domain; PP and DP across the scale-out fabric. A correctly mapped GB200 NVL72 job sets TP+EP to fit within the 72-GPU domain, runs PP across racks, and lets DP all-reduce ride the rail-optimized fat-tree where SHARP in-network reduction can absorb it. Get the mapping backwards — DP inside the domain, TP across racks — and you have inverted the bandwidth hierarchy: the cheap-to-cross dimension hogs NVLink while the expensive-to-cross dimension chokes on NIC bandwidth. This is why the scheduler needs more than a node count; it needs the parallelism shape, so block sizes and segment sizes line up with the TP/EP group rather than slicing through it. The framework that emits this shape is the subject of Chapter 10.8; the fabric that carries it is Chapter 8.5.

Topology-awareness creates a tension that does not exist in flat scheduling: the better you defend locality, the worse your bin-packing gets. If a job can only land inside one NVLink domain, then a domain with 6 of 72 GPUs busy has 66 GPUs that are useless to any job needing more than 66 NVLink-local — they are stranded by fragmentation, not by lack of capacity. Flat schedulers never see this problem because they will place a job anywhere; they pay for it instead in silent cliffs. Topology schedulers make the fragmentation visible and force a policy choice, which is the honest trade.

There are three levers, and a 2026 fleet uses all three. Accept the strand: keep domains whole, leave the 66 GPUs idle until a job that wants them arrives, and price the lost utilization as the cost of full-bandwidth placement — the right call when goodput per job dominates, i.e. large training. Defragment: use preemption and migration to compact small jobs out of partially-used domains, freeing whole domains for large ones — effective but expensive on synchronous jobs because preemption forces a checkpoint-and-restart. Relax locality with segments: let jobs declare smaller atomic groups (Slurm --segment, or sizing the ComputeDomain below a full rack) so they fit the holes you have, accepting scale-out traffic on the boundary the segment crosses. On rack-scale hardware the rack — the NVLink domain — becomes the natural unit of allocation, not the GPU and not the node. You schedule, bill, drain, repair, and reserve in units of domains, because any policy that subdivides a domain has to reckon with the IMEX-per-node constraint, the bandwidth cliff, and the fragmentation it creates.

The same mistakes recur because each comes from treating the GPU as fungible when the fabric says it is not:

• Flat scheduling on rack-scale hardware. Exposing nvidia.com/gpu: 8 with no clique or block awareness, then wondering why a 16-GPU job that landed 8-and-8 across two racks runs at 60% MFU. The hardware is non-blocking inside the domain and an order of magnitude slower across it; a location-free resource model cannot see the difference and so cannot avoid the cliff.
• Topology without gang scheduling. Placing the right pods on the right nodes but admitting them incrementally, so synchronous jobs deadlock holding partial allocations. Locality and all-or-nothing admission are a pair; shipping one without the other trades a silent cliff for an outright stall.
• Inverted parallelism mapping. Putting data-parallel all-reduce inside the NVLink domain and tensor-parallel across racks — using the scarcest bandwidth on the dimension that least needs it. The mapping must follow the appetite: TP/EP local, PP/DP remote.
• Trusting labels over measurements. Letting a node into the schedulable pool because it booted and its clique label is present, without verifying the NVLink bandwidth matrix — placing jobs onto a degraded domain the scheduler believes is healthy.
• Subdividing domains under the IMEX-per-node limit. Designing a Kubernetes multi-tenant scheme that hands partial nodes to distinct ComputeDomains, then discovering the one-IMEX-channel-per-node constraint strands the remaining GPUs. On rack-scale NVLink hardware, allocate in whole domains and bill in racks.