SGD · stochastic gradient descent
The iterative optimization loop underlying model training; whether it synchronizes every step or only intermittently determines how far apart training sites can physically sit.
Current numbers
~500xless inter-site communication for DiLoCo (8 workers) vs fully-synchronous SGD, matching convergence on C4