BF16 · Brain Float 16
A 16-bit floating-point format with a wide exponent range, a common default for stable AI training.
Current numbers
~2x / ~4xtensor throughput per step down the ladder (BF16→FP8→FP4), same silicon
~14 bytes/paramcheckpoint state per parameter (2B BF16 weights + 4B FP32 master + 4B+4B Adam moments); ~7x the weight file
~16 B/paramtraining-state footprint with mixed-precision Adam (2 BF16 weight + 2 BF16 grad + 12 FP32 master+moments); the number the framework must shard across the fleet