All-reduce
A collective operation that sums gradients across all GPUs and shares the result, the dominant traffic in training.
Current numbers
~30–94x / ~890–2,810xencoded-payload cut (message size × syncs) for SparseLoCo (H=15, 2-bit) vs 8-bit DiLoCo and 16-bit AdamW DDP, density-dependent (3.12% → 0.78%); at 8 workers, ring all-gather versus dense ring all-reduce divides these reduction factors by 4
~806–928 GB/s (2–32 GiB; sweep average ~872 GB/s)GB200 NVL72 in-domain all-reduce busbw across 2–32 GiB buffers (one provider's 72-rank run); scale-out gate set separately