observed interruption cadence for one Llama 3 405B training run on 16,384 H100s: 419 unplanned interruptions over 54 days, mean interval ~186 minutes
~186 minobserved
| Value kind | observed — Reported measurements, counts, and specifications keep the precision and scope stated by their source; an exact specification is not treated as a range. |
|---|---|
| Scope | Observed whole-job interruption cadence for the named 54-day run and its event definition. It is not a per-GPU hazard, an equipment AFR, or a scaling law for another fleet size. Apply checkpoint models only with the named project's measured whole-job interruption distribution and checkpoint costs. |
| As of | 2024 |
| Source | Meta (The Llama 3 Herd of Models, arXiv 2407.21783) |
| Review | checking…review by 2026-11-08 · standard cadence |
| Recorded changes | last 2026-07-02 · 2 revisions tracked |
| Claim id | mtti-for-a-405b-run-on-16k-h100-419-interrupts |
Where the guide uses it
← Full numbers register — every date-stamped figure in the guide, with revision history.