Scale-Up Fabric (Intra-Node / Intra-Rack)

Chapter 8.1 drew the three-network model (scale-up, scale-out, scale-across) and argued that the boundaries between them move with each silicon generation. This chapter lives inside the innermost ring. The scale-up fabric is the set of accelerators that communicate at memory speed: a switched or topology-specific peer interconnect running roughly an order of magnitude faster per device than the back-end NIC (about 8–18x per direction, depending on the NIC generation it is paired with). Across it, a tensor- or expert-parallel shard can be split without the collective collapsing. By 2026 the domain has climbed out of the eight-GPU server into named rack designs such as the 72-GPU GB200 NVL72, and NVIDIA’s March 2026 Rubin Ultra forecast reaches 576 GPUs across multiple racks, with its own topology and software model.

Watch the naming. At GTC 2025, “Rubin Ultra NVL576” meant one Kyber rack of 144 GPU packages, counting four dies each; at GTC 2026 the same name means eight 72-GPU racks joined by optics, and the single Kyber rack became NVL144. Kyber’s timing is contested: NVIDIA targets the second half of 2027, and SemiAnalysis reported in July 2026 that it had slipped to 2028. Chapter 16.2 keeps the dated profile.

Domain size is a decision with a cascade behind it, and no datasheet hands it to you. Too small, and you cap the tensor-parallel group that fits locally or push expert exchanges onto the slower back-end fabric; MoE inference throughput falls when those exposed exchanges miss the request deadline. Too large, and you pay for unused capacity and a wider drain unit that can take hundreds of GPUs out of service at once. Routes beyond their electrical budget add an optical bill, and choosing CPO on top adds its board-or-switch repair boundary and spares cost. Buy it from one vendor and you inherit their roadmap and their margin; assemble it from a standard and you inherit an integration burden and a maturity gap.

The defining property is per-device bandwidth and semantics, not topology or distance. NVLink 5 and NVLink 6 give 1.8–3 TB/s per accelerator in NVIDIA’s bidirectional-aggregate convention: 900 GB/s to 1.5 TB/s each way, or 14.4–24 Tb/s in network units. The scale-out NIC each platform ships with gives 50–200 GB/s per direction (400G to 1.6T). The gap is 18x on GB200 with a 400G ConnectX-7, 9x on GB300 with an 800G ConnectX-8, and about 7.5x on Vera Rubin with 1.6 Tb/s of ConnectX-9 per GPU. The NIC has doubled along with NVLink, so the gap has stayed near an order of magnitude, and it is the reason the two networks exist separately. Chapter 8.1 works the bytes-versus-bits and bidirectional-versus-per-direction conversions behind every comparison in this Part.

Inside the scale-up domain, accelerators address each other’s HBM with load/store or one-sided semantics and run collectives (all-reduce, all-gather, all-to-all) at a latency and bandwidth the back-end fabric cannot touch. The rule of thumb follows: fit your tensor parallelism and your expert parallelism inside the scale-up domain; let data and pipeline parallelism span the scale-out fabric. Cross that boundary with the wrong collective and you have moved a memory-bandwidth-bound operation onto a network roughly an order of magnitude slower.

Domain size is a workload variable, not a ladder of interchangeable maxima. An eight-GPU HGX group and a 72-GPU GB200 NVL72 rack put different amounts of resident model, expert state and workspace inside one local fabric. Count the devices the scheduler can actually allocate together before deciding which TP/EP exchanges must cross the back-end. Topology changes what the headline number means. AMD’s eight-GPU MI350-series node (MI350X or MI355X) has no switch: each GPU has seven Infinity Fabric links, one per peer, at 153.6 GB/s bidirectional each, 1,075.2 GB/s in total (AMD ROCm documentation). A balanced all-to-all uses all seven links, but a transfer between two GPUs gets one link by default, about a seventh of the headline. On an NVSwitch node any pair can use the full per-GPU bandwidth, so benchmark point-to-point and skewed traffic, not only balanced collectives.

Then check which operations work across that population: peer reads and writes, the atomic widths you need, memory ordering and the runtime’s collective path. Remote addressability, cache coherence and a collective API are separate requirements, and connectivity does not create a coherent shared HBM cache. NVLink-C2C CPU–GPU coherence, CUDA peer access and MNNVL/IMEX access domains solve different problems. A fast message-passing ICI system belongs in this comparison even though the application gets no general load/store address space. Record what makes a remote write visible and who revokes access after a process or device fails; without those rules, a larger addressable population is not a usable domain.

NVIDIA’s NVLink is the incumbent and the reference every challenger is measured against, so it anchors the discussion. The architecture has two parts: NVLink, the SerDes-based per-GPU link, and NVSwitch, the crossbar ASIC that turns point-to-point links into an all-to-all switched fabric. The per-GPU number is the generational headline: NVLink 4 (Hopper) delivered 900 GB/s, NVLink 5 (Blackwell) doubled it to 1.8 TB/s, and NVLink 6 (Vera Rubin) raises it to 3 TB/s, about 12x a PCIe Gen6 x16 link by NVIDIA’s count. NVSwitch makes a domain out of those links. In GB200 NVL72, nine NVSwitch trays wire 72 Blackwell GPUs into a single non-blocking domain with ~130 TB/s of aggregate NVLink bandwidth and ~13.4 TB of aggregate HBM in the GB200 NVL72 profile checked September 2026. Software accesses and synchronizes that memory through the supported programming model, not as one coherent cache. Vera Rubin NVL72 keeps 72 GPUs and nine switch trays and reaches 216 TB/s per rack on NVIDIA’s product page; production racks have shipped since June 2026, when Dell delivered the first to CoreWeave (2026-06-01), with Supermicro following in September (2026-09-23). NVIDIA’s January 2026 announcement gave 3.6 TB/s per GPU and 260 TB/s per rack, so confirm the figure on the purchase quote.

The NVL domain is the resource the placement and health services must describe. A GB200 NVL72 places 72 GPUs in one rack; keep that record attached to GB200, and do not use a successor’s package or die counts to enlarge it. The current NVLink Fusion platform page describes an NVLink 6 domain separately from its future domain-size roadmap. Multi-Node NVLink (MNNVL) and Internode Memory Exchange (IMEX) set up supported cross-node access and permissions; they do not turn every connected cache into a coherent copy. The domain is a first-class allocatable resource: a scheduler must intersect that access domain with available GPUs, validated routes and the job’s recovery unit. Allocate across a boundary that has not passed validation and a placement convenience becomes a repeatable communication or protection failure. Chapter 8.1 maps collectives to parallelism.

NVLink-SHARP is an in-domain capability to test alongside peer operations. Record the GPU and switch generation, collective library, operation, data type, message range, concurrent-group limit and fallback path, because a switch can expose the hardware while a particular call still runs on the endpoints. The byte-count comparison and the reduction acceptance method have one home, Chapter 8.6; here, decide whether the supported domain provides the capability the workload needs.

Buying an NVLink domain ties endpoints, switches and collective software to NVIDIA’s supported interface, a strong supplier dependency even when other local accelerator fabrics exist. The fork is strategic before it is technical. The accelerator, the switch and the collective library are co-designed around the scale-up domain, which makes it a major lock-in surface in the stack, and breaking that open is the challengers’ explicit goal. The 2025–26 picture is a three-way contest rather than a convergence. NVIDIA opened NVLink to other silicon through NVLink Fusion. AWS said in December 2025 that it is designing Trainium4 to use NVLink 6 and MGX racks; MediaTek, Marvell and d-Matrix plan custom accelerators on it, and Qualcomm and Fujitsu use it to attach their CPUs to NVIDIA GPUs. AMD and others back UALink and the Ethernet-based SUE/ESUN path.

NVLink / NVLink Fusion is the integrated or licensed-IP path. Fusion’s 2025-05-18 introduction enables custom endpoint integration, not arbitrary mixed-accelerator interoperation. UALink is an open specification. Its 200G 1.0 ratification announcement, 2025-04-08, describes reads, writes and atomics at 200 Gb/s per lane for domains of up to 1,024 accelerators; those are specification limits, and they do not prove a purchased system’s population or latency. Its 2026-04-07 family announcement covers Common 2.0, 200G DL/PL 2.0, Manageability 1.0 and Chiplet 1.0, with Common adding in-network compute. Each is a separate specification scope, so a purchased combination need not implement every option.

Ethernet scale-up (SUE/ESUN) reuses Ethernet SerDes, switch silicon and interfaces, so a merchant ASIC can potentially serve both scale-up and scale-out, but the endpoint transport must supply the required memory operations. ESUN, an OCP workstream launched in October 2025, covers switch-side framing, header optimisation and lossless delivery, and Broadcom’s SUE v1.0 specification (September 2025) adds link-layer retry (LLR) and credit-based flow control (CBFC). AMD identifies UALoE in its Helios platform; Ethernet carriage is not native UALink PHY conformance. Helios joins 72 MI455X GPUs through four scale-up cartridges of 200G Ethernet switch ASICs, for up to 260 TB/s aggregate by AMD’s projection. AMD spreads each GPU over 18 independent stations on different switch trays so that one failed path does not break the domain, and expects volume deployments in the second half of 2026. Broadcom’s 2026-03-12 Tomahawk 6 production-volume announcement establishes silicon status, not the endpoint and software support for any one rack.

The table is a bet on supplier power. The vertically integrated path buys an accelerator, NVLink switch and collective-software stack co-designed and supported as one combination. The price is single-vendor dependence at the most strategically important boundary in your cluster. The open paths invert that trade: you accept a maturity gap and an integration burden in exchange for multi-vendor switch and accelerator sourcing, and a credible threat that keeps incumbent margins honest. Even buying NVLink is a bet that NVIDIA’s roadmap stays far enough ahead, fast enough, to justify the lock-in premium. Chapter 8.3 frames the merchant-versus-captive silicon business model.

Two adjacent fabrics belong in the same mental model. Their programming and placement rules differ, and neither is a drop-in replacement for a chosen accelerator fabric. CXL (Compute Express Link) is cache-coherent over PCIe physical layers. Its sweet spot is memory expansion and pooling: adding or sharing CXL-attached DRAM across hosts (shipping expanders and pooling devices are DDR-based), not the terabyte-per-second all-to-all that training collectives demand. It is not a way to enlarge or pool an accelerator’s on-package HBM. In an AI rack, CXL and the scale-up fabric are complementary: CXL widens the memory pool and disaggregates capacity (increasingly relevant to KV-cache tiering), while NVLink/UALink carries the high-bandwidth collective traffic. Conflating them strands bandwidth on the wrong network; using CXL for an all-reduce, or a scale-up fabric for cold-capacity pooling, leaves performance on the table.

Google’s ICI takes a different scale-up design point. In a 3D torus each chip has six directional neighbors (±X/±Y/±Z), so collective traffic follows topology-dependent hops rather than an all-to-all crossbar. Keep Ironwood’s 2025-04-09 announcement as its own record. Google’s 2026-04-22 architecture account describes an 8t 3D torus and an 8i Boardfly system with different chip populations; these are platform topology records, not equal service slice sizes. Optical Circuit Switches (OCS) change physical connectivity without becoming packet switches. Reconfiguring around faults means a repair need not cost the same physical neighborhood, but the offered slice shapes and compiler placement decide what a job can exploit. Choose the service on the collective paths and recovery behavior of that slice, because a larger advertised population does not promise a shorter critical path. Chapter 8.5 returns to topology choices.

This is where domain size turns into MFU and tokens-per-dollar. For training, the scale-up domain sets the ceiling on tensor parallelism and the practical limit on expert parallelism. Tensor parallelism shards a layer’s matmuls across GPUs and exchanges activations on the forward/backward schedule you choose. That all-reduce is latency- and bandwidth-bound; keep it inside the scale-up fabric, where it stays cheap, whenever its exposed latency and bandwidth demand would exceed the next tier’s budget. Push TP past the domain boundary onto the back-end network and MFU falls hard, because the per-step collective now runs at scale-out speed. A 72-GPU domain lets a frontier model carry a high TP degree and still leave room for pipeline and data parallelism across the scale-out fabric. An 8-GPU domain holds only TP-8 on the memory-semantic fabric; a wider TP group still works but pays scale-out latency on every step, so budget any communication outside that group for the slower links it actually crosses.

For inference, the domain enables wide expert parallelism in Mixture-of-Experts models. Each MoE layer routes tokens to a subset of experts through an all-to-all, the most punishing collective for a slow fabric. NVIDIA measured up to 1.8x higher output throughput per GPU for EP32 than for EP8 on DeepSeek-R1 on GB200 NVL72, at 100 tokens/s per user (October 2025). It credits four factors: less weight loading per GPU, because each holds fewer experts; more efficient grouped matmuls; a dynamic expert load balancer; and the 130 TB/s NVLink domain, which keeps the wider all-to-all from eating the gain.

The domain also reshapes prefill/decode disaggregation. A large enough supported domain lets you place a prefill pool and a decode pool in the same NVLink island and stream KV-cache between them at memory speed rather than over the network, the substrate behind GB300/GB200 NVL72 + Dynamo MoE serving. Your maximum profitable EP degree, and with it tokens-per-dollar on large MoE models, depends on resident expert state, dispatch/combine time and recovery cost. A hierarchical implementation can cross scale-up domains when the measured savings justify the slower links. The inference archetype is in Chapter 1.3, the training archetype in Chapter 1.2.

Resident state is 16×60=960 GiB; sixteen devices reserve 16×64=1,024 GiB. Both placements fit, while one eight-device domain alone reserves only 512 GiB and fails. Dispatch bytes are 4,096×2×16×1,024=134,217,728 bytes; combine doubles the transfer to 268,435,456 bytes. In one domain, divide by 300×10⁹ bytes/s: about 0.9 ms, within 2 ms. Split evenly, half crosses the cut: 134,217,728/(45×10⁹) s is about 3 ms before any local-transfer delay, so it fails. Choose the 16-device domain and accept its larger scheduling and drain unit.

The expert count flips the answer at eight. With E≤8 and an unchanged 60 GiB per expert, E×60≤512 GiB, so one eight-device domain fits, and the assumed local service still finishes the same token traffic within the allowance. At E=9, 540 GiB exceeds 512 GiB, and the smaller domain fails on capacity before communication matters. If E exceeds sixteen, the 16-device domain also loses its one-expert-per-device placement: buy a larger supported domain, or test hierarchical EP with a larger time allowance or a faster cut. The NVIDIA Hybrid-EP design, 2026, supplies the mechanism, not these rates. Chapter 8.1 owns the dependency budget; Chapter 8.5 prices the cut.

Define three boundaries before allocating a domain: the peers authorized to map memory, the routes healthy enough to carry the job, and the resources that must drain for a repair. On GB200 with MNNVL, NVIDIA’s IMEX/Kubernetes design makes the access domain explicit, but it does not replace health-aware placement. A failed link first removes a path. Continue only if the platform supports that degraded state and the remaining paths still connect every placed rank with the required protection and time budget. A switch fault can remove several paths at once, so derive the surviving graph from that platform’s topology instead of multiplying a generic “GPUs affected” figure.

When that graph or the supported service procedure fails, stop new placement into the affected unit, quiesce surviving ranks, checkpoint only if the application can still produce a consistent checkpoint, revoke stale peer mappings and drain the required unit. If quiescence cannot complete, restart from the last valid checkpoint. After repair, revalidate link health, access controls and the same collective before returning the unit to the allocator. Count lost useful work and drain/restart time in Chapter 9.4; Chapter 13.7 proves the installed fault cases. A link count alone does not authorize service during live operation.

The physical gate is the validated electrical route. Package escape, host-board trace, connector transitions and cable length all consume its budget. GB200 NVL72’s compact copper spine keeps its in-rack NVLink routes electrical. Avoiding optical engines and lasers saves their power and removes their repair exposure, which makes copper a deliberate strength when the validated route fits. No port-rate label supplies a universal distance, so keep the exact endpoint generation, cable assembly, route and thermal limits that passed testing. When that route fails, compare a shorter layout, active copper and supported optics on delivered service and repair time. Chapter 8.9 owns the channel budget, so this chapter does not derive reach a second time.

The wall arrives when the domain’s routes outgrow copper’s validated reach. NVIDIA’s 2026-03-16 design account forecasts a 576-GPU Vera Rubin Ultra NVL576 domain across eight 72-GPU MGX racks with copper and direct optical connections. That named layout makes rack-to-rack optics part of its design; it is not a rule for every multi-rack domain. It is the reach argument for optical scale-up, while co-packaged optics (CPO) is a separate packaging choice. CPO moves optical engines alongside the switch ASIC, shortening electrical paths. Its power saving depends on the engine, laser and pluggable alternatives you match it against, and the saving multiplies with the installed link count.

The NVL576 design announcement establishes a future optical topology, not a shipment date for a production-qualified CPO NVSwitch. Broadcom’s 2025-10-08 Davisson announcement specifies early-access sampling. NVIDIA separately declared Spectrum-X Ethernet Photonics in production on 2026-05-31, an Ethernet status that does not establish NVLink optical availability. A domain whose routes exceed its copper budget commits the buyer to optics, but the reach decision and the packaging decision have different consequences. A supported span can use faceplate pluggables, a near-package engine or co-packaged optics. Moving the engine off the faceplate changes which failures need a module, board or switch replacement, and so changes repair time and spares. Chapter 8.10 engineers CPO and the fiber plant.

Larger domains are tightly coupled to power, but each platform sets its own planning boundary. NVIDIA’s NVLink platform page assigns 900 GB/s, 1.8 TB/s and 3 TB/s per GPU to Hopper, Blackwell and Vera Rubin (checked September 2026). That sequence is a set of platform specifications, not a law that every generation doubles useful collective throughput. Domain size follows the same distinction: the 2026-03-16 design account describes Vera Rubin NVL72 separately from future Rubin Ultra NVL576 across eight MGX racks and Kyber’s future NVL144/NVL1152 layouts. The medium moves from copper to optics where those named routes require it; the rack boundary alone does not select CPO. Power belongs to the same purchased profile: HPE’s GB200 and GB300 rack inputs, its VR200 cabinet basis and future Kyber power/distribution plans cannot be joined into one interchangeable rack specification.

Bandwidth, domain size and medium constrain one another, so a roadmap that treats them independently mis-sizes power, cooling and pathways. The multi-rack step also commits the buyer to a larger scheduling and recovery boundary. The future topology becomes a procurement option only when its endpoint, switch, software and repair combination passes testing by the project’s deadline. Space and pathways preserve that option, while an equipment order placed on a delivery forecast, before that combination passes testing, puts the deployment schedule at risk. Chapter 16.2 keeps the dated subsystem profiles, including future high-voltage distribution and optical-domain timing; Chapter 16.1 gives the macro power-bound rationale.

Choose the smallest supported domain that holds the placed state and meets the exposed-communication budget with its required operations intact. Pay for a larger domain when it removes a demonstrated deadline miss. Accept hierarchical crossings when the local capacity limit forces them and the tested cut can carry them. Get this wrong and you pay a predictable cost in idle accelerators, constrained placement or a larger restart.