Data parallelism · DP
Replicating the model across GPUs that each process different data and synchronize gradients via all-reduce.
Current numbers
100–500xless inter-site data exchanged by DiLoCo-class intermittent sync vs full synchronous DP — what makes cross-region training viable