MTTF · Mean Time To Failure
Average run time before a failure. For a training job it falls as the job grows: Meta reported about 47.7 days for 8-GPU jobs and 7.9 hours for 1,024-GPU jobs in its research clusters.
Current numbers
7.9 hrMTTF of a 1,024-GPU job vs 47.7 days for an 8-GPU job, observed on Meta research clusters
1.8 hr → 14 minModeled MTTF for hypothetical synchronous jobs of 16,384 and 131,072 GPUs: Meta’s observed A100 research-cluster failure rate, scaled as 1/n
~1.8 hr (projection)projected mean time to failure for a 16,384-GPU synchronous job: 1/(nodes × observed failure rate)