DiLoCo
Distributed Low-Communication training — sync-infrequently methods that tolerate WAN latency, letting one run span sites without a lockstep fabric between them.
Current numbers
~30–94x / ~890–2,810xtotal traffic cut for SparseLoCo (H=15, 2-bit) vs 8-bit DiLoCo and 16-bit AdamW DDP, density-dependent (3.12% → 0.78%)
~500xless inter-site communication for DiLoCo (8 workers) vs fully-synchronous SGD, matching convergence on C4