Training run — cost & time
Estimate a training run end to end: model parameters and tokens (6·P·T FLOPs), MFU, cluster size, and goodput → wall-clock days, GPU-hours, rental cost, facility energy, and emissions.
Training run — time, cost, energy & CO₂
MFU and goodput compound: a 40% MFU × 90% goodput scenario yields 36% of peak theoretical FLOPs as committed training work, by the definitions you use. Treat 90% versus 96% only as an illustrative sensitivity; measure your own fleet, job and time window, then find where the lost time went before blaming checkpointing or node cordoning → Ch 12.2.
Worked decision — stated assumptions
All ledger inputs are assumed teaching values chosen to exercise this chapter’s method; they are not market quotes, OEM performance claims or operating records. Arithmetic uses the full entered precision.
| Input | Value |
|---|---|
| Model size | 70 B params |
| Training tokens | 15 T tokens |
| Peak per GPU | 2.5 PFLOPS |
| Cluster size | 10000 GPUs |
| MFU | 40 % |
| Goodput | 90 % |
| Rental rate | 18 $/GPU-hr |
| GPU TDP | 1200 W |
| Host overhead | 1.5 × GPU TDP |
| PUE | 1.2 ratio |
| Power price | 0.08 $/kWh |
| Grid carbon | 400 gCO₂/kWh |
Illustrative dense-model run: assume 70 billion parameters, 15 trillion training tokens, 10,000 GPUs at 2.5 dense PFLOPS each, 40% MFU and 90% goodput. Define MFU on productive compute time so goodput does not deduct the same lost time twice. Training work is 6 × 70 billion × 15 trillion = 6.3 × 10²⁴ FLOPs. Effective rate is 10,000 × 2.5 × 10¹⁵ × 0.40 × 0.90 = 9 × 10¹⁸ FLOP/s. The run takes 700,000 seconds, or 8.10 days and about 1.94 million GPU-hours.
At an assumed $18/GPU-hour, rental is $35 million. With assumed 1.2 kW GPU power, a 1.50 total-IT multiplier and PUE 1.20, constant-power energy is 4,200 MWh: $336,000 at $0.08/kWh and 1,680 tonnes at 400 gCO₂/kWh. Do not add electricity to rental when the rental already includes it.
The run passes an illustrative nine-day deadline. At 80% goodput it takes 9.11 days and fails. Change the training-work model for sparse or non-dense workloads; the screening formula does not qualify model placement or convergence.
The nine-day deadline requires at least 81.0% goodput: 6.3 × 10²⁴ / (10,000 × 2.5 × 10¹⁵ × 0.40 × 9 × 86,400). Below that, reserve more capacity or move the deadline.
Primary method: Kaplan et al. (2020), §2.1 — 6N training FLOPs per token. Method and handoff: Chapter 12.2 — The AI-Cluster Reliability Rethink: Goodput vs Facility Availability.
These are transparent screening estimates, not final designs, financial advice, or project approvals. Replace every default with current project data and have the responsible project authorities validate the result. You can save, share a permalink, or export to CSV; see the full calculator suite.