Scale-Across: Multi-Campus & Cross-Region Fabric (DCI for Distributed Training)

Three chapters of this Part built the network inward-out: the scale-up domain that fuses 72–576 accelerators into one coherent memory machine (Chapter 8.4), and the scale-out Clos fabric that stitches scale-up domains into a 100k–1M-GPU cluster inside a single hall (Chapter 8.5). This chapter is the third tier of that hierarchy — scale-across — and it exists for one reason that has nothing to do with networking: you cannot energize a single campus fast enough. The binding constraint of the 2026 era is megawatts at the substation, not accelerators in the supply chain (Chapter 16.1). When a frontier run needs 1–2+ GW and the largest campus you can interconnect in the timeline tops out at a few hundred MW, you are forced to do something the synchronous-training orthodoxy spent a decade saying you must never do: split one job across buildings, then across campuses, then across regions.

This is the defining frontier-lab architecture of 2025–26. Google trained Gemini Ultra across multiple datacenters on TPUv4 pods, combining SuperPods over its intra- and inter-cluster network with latency and bandwidth sufficient to keep the run synchronous. OpenAI/Microsoft and the Stargate program are building gigawatt-class liquid-cooled campuses explicitly to chase the same multi-datacenter training capability. Scale-across is a chain of forced forks: the moment you accept that the power won't fit in one building, you have implicitly committed to a latency floor, a class of optics, a transport layer, a training algorithm, and a new and nastier failure model. Each fork below carries a goodput cost for choosing it wrong.

The single most useful mental model for this whole Part is the three-tier hierarchy, and scale-across is the tier where the numbers fall off a cliff. Inside the scale-up domain, NVLink delivers ~1.8 TB/s per GPU (NVLink 5) at roughly 100 ns latency — load/store and collective semantics, one giant GPU. Scale-out drops to ~400–800 Gb/s per NIC at ~1–2 µs over an InfiniBand or Ethernet Clos. Scale-across drops again, by orders of magnitude on both axes at once: per-site uplink bandwidth is whatever DWDM you can light and pay for (tens of Tb/s aggregate is a large buy), and the latency floor is set by physics you cannot engineer around — ~5 µs per kilometer of fiber, one way.

That 5 µs/km figure is the gravity of this chapter. Light moves at ~300 m/µs in vacuum but ~30% slower in silica (refractive index ~1.47), so standard SMF-28 fiber adds ~4.9 µs/km one-way — call it 5. A campus-to-campus link 40 km apart has a ~200 µs one-way floor, ~400 µs round trip, before a single transponder, amplifier, FEC block, or switch hop adds its own delay. Two regions 1,000 fiber-km apart sit at ~5 ms one-way, ~10 ms RTT. Compare that to the ~100 ns of NVLink: scale-across is four to five orders of magnitude slower than scale-up. An all-reduce that completes in microseconds inside the hall takes milliseconds across the WAN, and a synchronous training step that spends a meaningful fraction of its time blocked on that all-reduce sees its MFU collapse. The entire algorithmic apparatus of this chapter — relaxed synchrony, gradient compression, hierarchy — exists to hide or amortize that floor.

Before any optics or algorithm, the honest first question is whether you are actually power-bound. Going multi-site imposes a permanent goodput tax — best case you lose a few percent to inter-site synchronization overhead and operational complexity; worst case a naive synchronous split halves your effective throughput. You take that tax only because the alternative — waiting for a single campus to be energized — costs you more. The math is a race between two clocks: the depreciation clock on accelerators you've already bought (2–3 year economic life; idle silicon is pure loss) versus the interconnection clock on the megawatts you need (4–7 year large-load waits in the densest US hubs). When the power clock is slower than the silicon clock, you go multi-site. That is the whole rationale, and it is a power-bound rationale, not a networking one. → the macro narrative lives in Chapter 16.1; siting and queue mechanics in Chapter 3.2.

The second-order question, once you've decided to split, is what the second site is for. There are two very different answers and they buy very different amounts of fiber. A training peer site runs part of the synchronous (or relaxed-sync) job and must exchange gradients or pseudo-gradients with the primary on a cadence that the algorithm sets — this is a bandwidth-hungry, latency-sensitive relationship. A checkpoint-and-burst replica holds a copy of model state and absorbs elastic inference or batch overflow — it needs enough bandwidth to ship checkpoints (gigabytes to terabytes on a slow cadence) but is indifferent to per-step latency. Mis-classifying the relationship is a classic over-spend: provisioning peer-grade DWDM for a site that only ever needed checkpoint-grade bandwidth, or the reverse — starving a true training peer and watching the run stall on a link you under-bought.

Inside the hall, the interconnect debate is copper-versus-optics over single-digit meters (Chapter 8.9). Across sites, copper is irrelevant and the debate is which class of coherent optics carries traffic over kilometers of glass. The fork has three branches, and they trade reach, cost-per-bit, and how much transport plumbing you own.

Grey optics over dark fiber is the intra-metro answer: a standard 800G or 1.6T pluggable (DR/FR-class) lit directly over a dedicated dark-fiber pair you own or lease IRU on, no DWDM line system. Cheapest per link, simplest operationally, but reach- and fiber-count-limited — every wavelength burns a fiber pair, and you run out of pairs (or money) past a campus boundary. Coherent ZR/ZR+ pluggables are the metro-to-regional workhorse and the defining DCI technology of this cycle. The OIF's 800ZR Implementation Agreement (October 2024) defines an 800G coherent line interface for single-span amplified 80–120 km DWDM — a router/switch port speaks DWDM directly, collapsing a whole rack of external transponders into a QSFP-DD/OSFP module. OpenZR+ / 800G ZR+ extends modes from 120 km DCI out beyond 1,000 km of regional transport over existing DWDM. Full OTN / managed transport is the heavyweight branch: a separate optical-transport network with its own protection, OAM, and capacity engineering — the right call when you need carrier-grade resilience, sub-lambda grooming, or you are leasing capacity rather than owning fiber. → the physical-layer primitives (modulation, FEC, link budgets, baud rates) are in Chapter 8.9; the fiber plant and structured cabling in Chapter 8.10.

This is the decision the rest of the chapter orbits. Once the job spans a metro or regional radius, you must choose how synchronization happens across the thin WAN pipe, and the choice trades convergence quality and simplicity against inter-site bandwidth and tolerance to latency.

Branch A — synchronous scale-across. Treat the remote campuses as one extended fabric: data-, tensor-, and pipeline-parallel groups span sites, and every step still ends in a global all-reduce that now traverses the WAN. This preserves exact synchronous-SGD convergence (the math is identical to single-site training, so model quality is uncompromised), and it is what Google did for Gemini Ultra. The cost is acute sensitivity to inter-site bandwidth and latency: you must overlap communication with computation aggressively, place the parallelism dimensions so the chattiest collectives (tensor-parallel) stay inside a site and only the more tolerant ones (data-parallel, pipeline) cross the WAN, and accept that bisection bandwidth across sites is your hard throughput ceiling. Get the placement wrong and the run is bandwidth-bound on the WAN, MFU in the teens. This branch is viable mainly intra-metro and at the easier end of metro.

Branch B — relaxed-synchrony (local-SGD family). Stop synchronizing every step. Each site runs many local optimizer steps independently, then the sites exchange and average their accumulated updates — "pseudo-gradients" — only periodically. This is the DiLoCo paradigm (Distributed Low-Communication): inner optimizer AdamW running H local steps, outer optimizer Nesterov momentum averaging across workers. The headline result is staggering — DiLoCo on 8 workers matched fully-synchronous optimization on C4 while communicating ~500x less, and OpenDiLoCo reproduced this training across two continents and three countries at 90–95% compute utilization. Streaming DiLoCo (2025) overlaps the periodic communication with compute to cut peak bandwidth further; follow-on work reports up to 16x less traffic than DiLoCo and up to ~3000x less than standard DDP. The cost is no longer bandwidth — it's convergence risk: relaxed synchrony introduces staleness and can perturb final model quality, the inner/outer optimizer and sync-interval are extra hyperparameters that must be tuned, and the failure modes are subtler than synchronous SGD's clean restart-from-checkpoint.

Branch C — hierarchical / async hybrids. The pragmatic middle most large multi-campus runs actually land on: synchronous within a site, relaxed across sites. Each campus is one tightly-coupled synchronous fabric; the campuses themselves are coupled with local-SGD-style periodic averaging or bounded-staleness asynchrony. This matches the algorithm to the physics tier-by-tier — pay full synchronous cost only where bandwidth is cheap (intra-DC), and pay the relaxed-sync convergence tax only where bandwidth is expensive (cross-WAN). Gradient/pseudo-gradient compression (quantization, top-k sparsification, low-rank/momentum decoupling) layers on top of any branch to shrink what crosses the WAN further.

Intra-DC reliability engineering assumes failures are independent: a GPU dies, a NIC flaps, a cable goes bad — uncorrelated events you ride out with hot spares and checkpoint-and-resume (Chapter 12.2). Scale-across breaks that assumption. A fiber cut between campuses, a regional grid event, a utility curtailment, a metro-wide cooling-water issue, or a site evacuation takes out an entire site's worth of accelerators at once — a single correlated failure spanning thousands of GPUs. The DCI link itself becomes a first-class fault domain: if the inter-site fiber is a single un-diverse path, one backhoe ends the run. The reliability posture that worked at flat intra-DC scale is no longer sufficient.

This reshapes three things. Path diversity stops being optional: inter-site fiber must be physically diverse (separate conduits, separate entrances, ideally separate carriers) or the WAN link is a guaranteed correlated-failure single point. Checkpoint placement must become site-aware: a checkpoint that lives only on the campus that just lost power is no checkpoint at all. The relaxed-synchrony algorithms help here almost for free — because each site already holds a recent local model copy between syncs, a site loss degrades to resuming from the last global average rather than a total restart, which is one of the underappreciated operational virtues of the DiLoCo family over naive synchronous scale-across. Blast-radius accounting must treat "site" as the failure unit: the question is no longer "what happens when a node fails" but "what fraction of the run survives when a whole campus drops, and how fast can the remaining sites continue or resync." → the checkpointing math — interval, bandwidth, and the storage/network trade behind multi-region durability — is worked in Chapter 9.4; the goodput-vs-availability reframing in Chapter 12.2.

The chapter resolves to an ordered sequence of forks, each one constraining the next, now stretched across the WAN.

• Are you power-bound on one campus? If no, stay single-site and skip all of this; the goodput tax of going multi-site is real and unrecoverable. If yes, the silicon clock is beating the power clock and you go across. → Chapter 16.1.
• What radius can you energize? Intra-metro, metro, or regional — set by which substations you can interconnect, not by preference. The radius picks your latency floor at ~5 µs/km. → Chapter 3.2.
• Which DCI optics and transport? Grey 800G/1.6T over dark fiber intra-metro; 800ZR coherent over DWDM for 80–120 km; OpenZR+/OTN for regional. → Chapter 8.9, Chapter 8.10.
• Synchronous or relaxed-synchrony? Synchronous only if the radius lets a per-step all-reduce fit the compute budget; otherwise hierarchical/DiLoCo-class with compression. This is the model-quality-vs-bandwidth fork. → algorithms above.
• How do fault domains and checkpoints span sites? Route-diverse fiber, site-aware checkpoint placement, blast-radius measured per site. → Chapter 9.4, Chapter 12.2.

Walk that sequence and the multi-campus run is a chain of well-posed decisions, each with a named downstream cost. Skip the first fork — go multi-site when you weren't actually power-bound — and you pay a permanent goodput tax for no reason. Skip the algorithm fork — run synchronous across a regional radius — and you light expensive fiber to watch both campuses idle on an all-reduce physics won't complete in time.