InfiniBand vs Ethernet (RoCE / scheduled fabrics)
The back-end fabric question is no longer whether Ethernet can do it. NVIDIA reports ~95% data throughput for xAI's Colossus on Spectrum-X Ethernet with adaptive routing and congestion control, and Meta tuned its RoCE and InfiniBand GenAI clusters to equivalent performance. The choice is now about supply chain, operations, and how much you pay for a fabric that works with little tuning.
| Axis | InfiniBand | Ethernet (RoCE / scheduled fabrics) |
|---|---|---|
| Effective throughput | credit-based and lossless by design, so collectives run well with little tuning; no common-workload test shows a universal lead over tuned Ethernet | adaptive-routing fabrics report ~95% data throughput (Spectrum-X at Colossus, vendor-reported); scheduled fabrics are a separate category; plain RoCE needs careful congestion tuning (DCQCN) to match |
| Ecosystem | one vendor (NVIDIA, Mellanox lineage): tight, closed, proven | many vendors: Broadcom Tomahawk and Jericho, the Ultra Ethernet Consortium, every switch OS |
| Cost per port | premium, with captive optics and cables | merchant-silicon pricing; 102.4 Tbps ASICs with high radix |
| Radix and scale | proven at large scale on a managed subnet | the highest-radix ASICs ship for Ethernet first, so fabrics get flatter with fewer tiers |
| Operations | a separate skill set and toolchain from the front-end network | one operating model for the whole estate, in the language your network team already speaks |
| Lock-in | high: fabric, NICs, optics and GPUs from one vendor | lower, though scheduled-fabric features can bring back vendor coupling |
| Where it wins | training pods bought as one validated vendor stack, where integration effort matters more than port price | cost at scale, multi-vendor strategy, inference fleets, and any team already running Ethernet, provided the acceptance test runs the real collectives at scale |
Quantitative cells are the guide's canonical figures — each is date-stamped and sourced in the numbers register and derived in the chapters below.
How the decision falls
Default to Ethernet for a new back-end fabric. Tuned well, it matches InfiniBand in published deployments (~95% at Colossus, vendor-reported), it runs on the network skills you already have, and merchant silicon keeps a second switch supplier in play. Choose InfiniBand when the cluster has to come up this year at frontier scale with the least integration work, and you accept one vendor's fabric, NICs and optics as the price of a single support boundary. Either way, run your acceptance test on the real collectives at full scale: no published test names a cross-vendor goodput winner (Chapter 8.4).
What would flip it: If you lack the team or the time to tune and run RoCE at scale, InfiniBand's integrated stack wins. Ultra Ethernet silicon maturing early, or NVIDIA pricing Spectrum-X hard against its own Quantum line, pushes the choice further toward Ethernet.
Model this fork with your own numbers: Cluster sizing calculator →
Full derivations, worked examples, and the numbers behind this matrix: Scale-out fabric: protocols and transport (Ch 8.4) · Network silicon: switch ASICs, NICs and DPUs (Ch 8.3) · AI traffic characterization (Ch 8.1)