InfiniBand vs Ethernet (RoCE / scheduled fabrics)
The back-end fabric question is no longer 'can Ethernet do it' — xAI's Colossus runs ~200k GPUs on scheduled Ethernet at ~95% effective throughput. It is now a systems decision about ecosystem, operations, and how much you pay for determinism.
| Axis | InfiniBand | Ethernet (RoCE / scheduled fabrics) |
|---|---|---|
| Effective throughput | the historical benchmark for collectives; credit-based, lossless by construction | scheduled/telemetry fabrics reach ~95% effective (Spectrum-X at Colossus); plain RoCE needs careful tuning |
| Ecosystem | single-vendor (NVIDIA/Mellanox lineage) — tight, closed, proven | multi-vendor: Broadcom Tomahawk/Jericho, UEC consortium, every switch OS |
| Cost per port | premium; captive optics/cables | merchant-silicon economics; 102.4 Tbps ASICs, 512×200G radix |
| Radix / scale | proven at large scale; managed subnet | highest-radix ASICs are Ethernet-first; flatter fabrics, fewer tiers |
| Operations | separate skill set + tooling from the front-end network | one operational model for the whole estate; your NetOps already speaks it |
| Lock-in exposure | high — fabric, NICs, optics, and GPUs from one vendor | lower — but scheduled-fabric features can re-introduce vendor coupling |
| Where it wins | maximum-determinism training pods inside one vendor stack | cost-at-scale, multi-vendor strategy, inference fleets, anyone already Ethernet-native |
How the decision falls
Training pods bought as a single NVIDIA system still default to InfiniBand or NVLink-adjacent fabrics; everything else is converging on Ethernet — and the UEC roadmap plus merchant-silicon economics keep pushing the crossover point down. Decide on the silicon layer first (SerDes, buffers, NIC offloads); the protocol fight is downstream of it.
Full derivations, worked examples, and the numbers behind this matrix: Network silicon: the gating layer (Ch 8.3) · Scale-out back-end fabric design (Ch 8.4) · AI traffic characterization (Ch 8.1)