SDC · Silent Data Corruption
Errors that corrupt computation without any alert; at fleet scale they silently spoil training and must be hunted.
Current numbers
every 1–2 weeksexpected SDC events during a large-scale training run (Meta; Google reports similar for Gemini)
~1 in 1,000machines harboring an SDC-prone defect; SDC expected every 1-2 weeks in large training
~2.5 billionSDC test seeds per month across Meta's fleet (Fleetscanner + Ripple)