DiLoCo
Distributed Low-Communication training — sync-infrequently methods that tolerate WAN latency, letting one run span sites without a lockstep fabric between them.
Current numbers
100–500xless inter-site data exchanged by DiLoCo-class intermittent sync vs full synchronous DP — what makes cross-region training viable
~30–94x / ~890–2,810xtotal traffic cut for SparseLoCo (H=15, 2-bit) vs 8-bit DiLoCo and 16-bit AdamW DDP — derived from Table 1 message sizes × sync counts; density-dependent (3.12% → 0.78%)
~500xless inter-site communication for DiLoCo (8 workers) vs fully-synchronous SGD, matching convergence on C4