All-gather
A collective that assembles each GPU's data shard onto every GPU, common in sharded training and inference.
Current numbers
~30–94x / ~890–2,810xencoded-payload cut (message size × syncs) for SparseLoCo (H=15, 2-bit) vs 8-bit DiLoCo and 16-bit AdamW DDP, density-dependent (3.12% → 0.78%); at 8 workers, ring all-gather versus dense ring all-reduce divides these reduction factors by 4