Checkpoint
A periodic save of model weights and optimizer state so a long training run can resume after a failure.
Current numbers
2.45–14 TB (exact assumed layout)Exact constructed checkpoint layout: 175 billion × 14 B = 2.45 TB; 1 trillion × 14 B = 14 TB
age of newest surviving captured statetraining lost-work boundary: captured progress in the newest valid checkpoint whose durable copy and recovery dependencies survive the declared event
restore <1 mincheckpoint restore latency with multi-tier checkpointing on Google's 35K-chip TPU v5p run — one clock inside MTTR, not MTTR