FP32 · 32-bit floating point
Single-precision format, the top of the precision ladder; mixed-precision training keeps master weights and optimizer moments here even when the math runs lower.
Current numbers
~14 bytes/paramcheckpoint state per parameter (2B BF16 weights + 4B FP32 master + 4B+4B Adam moments); ~7x the weight file
~16 B/paramtraining-state footprint with mixed-precision Adam (2 BF16 weight + 2 BF16 grad + 12 FP32 master+moments); the number the framework must shard across the fleet