Optimizer state
The extra per-parameter data an optimizer like Adam keeps (momentum, variance), often doubling or tripling memory needs.
Current numbers
~14 bytes/paramcheckpoint size incl. optimizer state — 100B ~1.4 TB; GPT-3 175B ~2.45 TB; 1T params ~14 TB; frontier runs sustain on < 1 TB/s global checkpoint BW
~14 B/paramrule-of-thumb checkpoint size on disk (weights + optimizer state); sets async-drain bandwidth need
~2.45-13.8 TBcheckpoint size for a 175B to 1T-param model at ~14 bytes/param incl. optimizer state — the at-rest bulk the crypto must wrap