Failure-Mode / FMEA Catalog

This appendix consolidates the failure modes scattered across the engineering chapters into one uniform FMEA register. It is referenced from the resilience-standards chapter (Chapter 12.1), the reliability rethink (Chapter 12.2), the component-failure-rate chapter (Chapter 14.3), and the integrated-systems-test chapter (Chapter 13.6); and it supplies candidate top events and common-cause couplings to the quantitative availability model in Chapter 12.5, after the project validates equipment, operating state, fault domain and evidence; these rows are not executable model inputs. The canonical engineering of each mode lives in the chapter named in the right-most column — this catalog is the index and the cross-walk, not the derivation.

Some random and adversarial scenarios can converge on a similar top event or protective response, but they do not automatically share one record. Keep initiating evidence, likelihood, authority, observability, persistence, propagation, security controls, and recovery distinct. Where the validated topology joins the paths, link the records to the shared top event and reuse the proven protective response; do not claim every random failure has an attacker twin. The cyber-physical analysis is in Chapter 11.10.

Each failure mode is recorded against six fields, applied identically across the three master tables so the catalog is sortable and comparable:

• Trigger — the initiating event (random cause first; adversarial initiators stay in separate Part 11 records linked to a proven shared top event).
• Propagation path — how the fault spreads, and critically which shared mitigation it defeats to escalate from a local fault to a cascade.
• Detection — the sensing modality and its characteristic latency relative to propagation speed (the decisive ratio).
• Blast radius — the fault domain affected at full propagation: node, rack, row, hall, or campus/POI.
• Mitigation — the preventive or containing control, classified as preventive (stops the trigger), inertial (buys ride-through time), or reactive (acts after detection).
• Recovery — the path back to service, kept as two separate numbers: service-restoration time (detect, isolate, transfer, repair or replace, restart, validate) and, where the mode costs work, the replayed work plus any integrity look-back to a known-clean state. Sum them only where they occupy distinct, non-overlapping portions of the job timeline; they are not interchangeable.

Where propagation outruns detection-plus-actuation, that reactive loop is too slow: prevent the trigger, add inertia, or qualify a faster independent protection path. The ratio matters because neither an operator nor an untested standby earns timing credit.

A direct-to-chip loop has two clocks. With flow maintained, the coolant and metal can buffer a heat step for tens of seconds where the measured mass and load support it. With flow lost, local mass remains but circulation no longer transports heat away; the rack's throttle and trip states can arrive in seconds, on a window the OEM's transient data and a commissioning pump-drop test set for the specific product. The leak and CDU/pump rows are engineered against the second clock: detection and switchover must complete inside it, which is why standby pumps switch over automatically — on UPS-backed feeds where the ride-through study shows the source transfer outlasts the window — and why GPU power capping, not operator action, is the last layer. The cause-and-effect is engineered and tested in Chapter 5.11; the measured response feeds the reliability model in Chapter 12.2.

Adversarial initiators: prove the shared physical event. Keep each security scenario’s access, command authority and persistence separate from random-fault rates. F-R01: spoofing a dew-point setpoint can force condensation where the command reaches exposed surfaces below dew point; the coolant-leak trace below retains that mechanism. F-R02: malicious VFD firmware or controller denial of service can stop circulation only where the compromised path can defeat the required flow and its independent protection. F-R04: CDU disablement that holds flow at zero links to HBM overtemperature only through the selected package’s demonstrated thermal response. F-R05: malicious power-cap firmware can force a synchronized load step only over the endpoints it controls; use the measured MW step and timing. F-R07: BMS spoofing that disables cell balancing or thermal protection reaches the battery hazard only where independent cell/pack protection does not veto it. Chapter 11.10’s OT destructive-primitive and control-dependency assessment owns these five initiators.

F-R09: a deliberate cut of an undiverse fiber route belongs to Chapter 11.2’s fiber-vault and sole-path sabotage assessment; the shared physical event is the lost route, not a new random cut rate. F-R10: a thermal attack on the optics environment is a candidate 11.10 OT scenario only if cooling authority or physical access can drive the named transceiver outside its qualified environment and link/error tests reproduce the flap mechanism. F-R11: fault injection on a known-marginal device is a candidate scenario only if the attacker controls a relevant voltage, clock or thermal path and isolated known-answer tests establish undetected wrong results; a crash or corrected ECC event is not SDC. 11.5 bounds the confidential-computing threat model; 11.10 assesses reachable controls and 14.3 owns SDC evidence. No attack frequency is inferred from component failure data. Credit protection only after it survives the assessed attacker authority.

Detection differs by mode: an active fiber-path cut alarms immediately at the physical layer, while an untested route-diversity defect and SDC remain latent; optics degradation is trend-detectable, and water curtailment is detected through make-up-water telemetry or receipt of the curtailment order. An SDC event can poison a checkpoint and sit undetected for days; an optics flap can erode MFU before tail latency is traced to a marginal transceiver; an untested protection path is exposed when the active route is cut. Match the control to the mode: a dedicated SDC-hunting program, per-port flap/FEC telemetry, verified-clean checkpointing, and periodic protection-path and physical-diversity verification bound the respective blast radii. The empirical fault taxonomy and the AFRs that feed these rows into the availability model are the canonical content of Chapter 14.3; the SDC detection program is treated there and demonstrated at commissioning in Chapter 13.6.

For the availability model, the catalog's most useful output is the couplings between modes — the shared resources whose failure makes two independent-looking modes fail together. These are the common-cause terms an RBD or fault tree must capture, or it will badly over-state availability. The table below names the cross-mode interlocks worth modeling explicitly.

The rows above are inputs to the reliability work, and each field maps to a model term. Top events come from the project's loss criteria and confirmed fault domains — a row becomes a top event when its blast radius reaches a domain the contract prices. Basic-event rates may take the fleet evidence in Chapter 14.3 where component, population, maturity and duty match; otherwise use project evidence and carry the uncertainty. Shared events come from the coupling table, confirmed against the as-built topology — a confirmed coupling is an explicit shared event, never an assumed beta. Recovery distributions use measured detection, protection, repair, restart and validation times; checkpoint cadence couples the training rows' recovery. IST cases are the rows the design basis names for demonstration, by the evidence method Chapter 13.6 sets; the catalog does not itself authorize a hazardous live event. The model mechanics are in Chapter 12.5.

F-R02.1, normal duty: a pump bearing fails while standby control and power survive independently. Flow loss exposes four racks to throttle/trip and job interruption. The budget is 0.50 + 0.50 + 3.0 + 1.0 = 5.0 s, leaving 1.0 s before the assumed 6.0 s limit. HOLD hardware acceptance: obtain the selected rack’s allowable interruption and qualify the whole response; the assumed six-second screen authorizes no live pump-drop test. The commissioning lead records approved test/equipment IDs, heat load, inlet state, minimum worst-branch flow, timestamps, uncertainty and restart evidence. Accept only with restored qualified flow before the limit and temperatures inside the OEM envelope; report lost work separately from service restoration.

At an exact nominal 4.0 s standby recovery the budget becomes 6.0 s: equality fails. Shorten recovery or change the protection/thermal design and retest. F-R02.2, shared controller lost: the same pump count earns no standby credit when one controller disables both trains; the four-rack consequence remains. The design lead owns independence correction; the commissioning lead keeps acceptance open until signed evidence demonstrates the protective function. Chapter 5.11 owns hydraulic protection, Chapter 13.6 test authorization, and Chapter 12.5 the evidence-fed model.