Level 5 Integrated Systems Testing (IST) & Failure-Mode Demonstration

Every level of commissioning before this one tested a subsystem in isolation against its own design intent: the switchgear in Chapter 13.3, the generators and microgrid in Chapter 13.4, the cooling plant and CDUs in Chapter 13.5. Level 5 Integrated Systems Testing (IST) verifies the integrated facility against the defined design-basis scenarios. Each scenario runs as a script — preconditions, witnesses, hold points, abort limits, expected result, restoration — and each design-basis fault has a valid evidence method: installed controlled tests on load banks, injection/HIL and verified analysis close different links, and none silently substitutes for an untested installed interface. IST is not authority to improvise concurrent faults beyond the design basis, or to put revenue-bearing GPUs behind the test.

And it is built on a lie of convenience. IST is run against load banks — heaters that consume the design power so the building has something to cool and power while you trip its redundancy. Load banks are how you can pull the plug on 100 MW without owning 100 MW of irreplaceable accelerators. But a load bank is a fundamentally different electrical and thermal object than a GPU cluster. A bank held flat draws a smooth, steady, controllable load; switched and programmable banks add bounded transients, while a GPU fleet supplies its software-synchronized, millisecond-scale waveform. That gap gets its canonical treatment here. After the IST master sequence and the failure-mode demonstration, the sections below set out what each load-bank technology can and cannot reproduce, across both the electrical and the thermal dimensions, and how a staged, production-representative run in Chapter 13.9 closes workload-specific normal-operation evidence that facility emulators cannot fully reproduce.

IST does not begin until every feeding subsystem has its own L4 acceptance signed and its baseline fingerprint captured (Chapter 13.2). The reason is diagnostic, not bureaucratic: if a cooling pump fails during defined-boundary loss-of-source or transfer evidence, you must already know it passed its standalone test, or you cannot tell whether IST found an integration fault or merely an un-commissioned component. IST tests seams, not parts. Sequencing it after the parts pass is what makes a failure during IST interpretable.

The master sequence is a planned escalation. It typically runs steady-state proving first (the building holds design load at design conditions for a sustained soak), then single-fault scenarios (lose one utility feed, one generator, one UPS module, one CDU), then the utility-loss transfer chain — the feed opened at the approved boundary with the building on load banks, and the ride-through, generator start and mechanical-plant transfer all timed — then the combined-fault matrix the design basis specifies, which is what proves the redundancy topology actually delivers the tier it was sold as. Build the IST planning and execution window from fixture mobilization, required states, witness availability, thermal stabilization, battery recharge and retest dependencies; observe these durations on the accepted equipment rather than importing a generic weeks-to-months allowance. The deliverable is a witnessed, time-synchronized data record of how the integrated building behaved at every transition, not a pass/fail stamp.

Attach a capability receipt to each rung: fixture serial/configuration; kW and kVA/PF; step size, slew, repetition and harmonic spectrum; liquid/air heat split; branch placement; control bandwidth; instrument uncertainty; and the exact failure interface injected. A load-step claim without a measured waveform is as weak as a power rating without a terminal. Avtron's generator load-testing method includes switched-load testing; Chapter 13.5 applies the independent liquid-path requirements. A missing capability means another fixture or an explicit HOLD, not an unseen test accepted by its label.

A resilience claim is a claim about its named design-basis states, and IST is where it is cashed. A concurrently maintainable design is proved by taking each path out of service in turn with the load held; a fault-tolerant design by tripping each single fault with the load held; and the combined faults the design basis names — a utility loss during a planned maintenance isolation is the classic — are tripped in the order most likely to expose a shared dependency. 2N promises survival of the design-basis events, not of an arbitrary second fault improvised on the surviving path, so the hidden single points of failure — two nominally independent systems sharing a breaker, a controller, a cable tray, or a cooling loop — are hunted by tracing the as-built topology and by relay secondary injection, then confirmed by the approved scripted trip where authorized, or by explicitly scoped injection/HIL and installed-interface checks, with any untested response left open. The cascading-fault matrix is built directly from the facility FMEA (the consolidated catalog lives in Appendix F): each high-severity failure mode becomes an IST scenario, executed in the order most likely to surface a shared dependency.

For AI factories, the dominant integration risk is no longer electrical — it is thermal ride-through, the seam that did not meaningfully exist in legacy IT. In an air-cooled hall, a brief cooling lapse during power transfer is absorbed by the thermal mass of the room and the air; you have minutes. In a direct-to-chip liquid hall, the thermal mass at the die is almost nothing. A GB200-class rack accepts coolant inside the envelope its own vendor documentation names — QCT's QoolRack reference gives separate maxima of ~45 °C liquid inlet and ~65 °C liquid return, and ~25 °C was NVIDIA's launch operating example, not a floor (Chapter 13.5); ASHRAE W45 describes facility-water supply capability, not a product setpoint. Past that envelope the GPUs throttle, and a sustained loss of flow reaches the throttle in seconds. How deep the throttle goes and how long an excursion may last are properties of the named server, coolant profile and workload — read them from the product documentation and measure them in the approved test rather than carrying a generic percentage into the script. So the IST question is specific: during the worst-case power transition, does coolant flow and temperature stay inside the envelope, or does the cluster throttle or trip? Critical cooling therefore rides on the protected bus, and its re-energization time is measured against that window, not assumed. IST is where you measure that gap against a stopwatch, with a load bank standing in for the heat the GPUs would have produced. This is the explicit interlock between facility Cx and cluster burn-in flagged in Chapter 13.5.

The control and monitoring stack is one of the things under test, not a witness to IST. Three layers must be proven to agree: the BMS (mechanical/electrical building automation), the SCADA / power-management system (switchgear, generators, the microgrid controller of Chapter 13.4), and the DCIM that operations will actually watch on day 2 (Chapter 14.2). The classic IST finding is not a hardware failure at all — it is that the building did the right thing while the DCIM displayed the wrong thing, or raised forty alarms for one event, or missed the event entirely because a Modbus/BACnet mapping was transposed during integration. An alarm flood is itself a failure: an operator who cannot find the root-cause alarm under a cascade of consequential ones will mis-diagnose the next real incident. IST validates the alarm hierarchy, the automatic control sequences (not just manual operation), and the point-to-point mapping from physical sensor to operator screen — the data path that the entire day-2 reliability program (Chapter 14.1) is built on top of.

The central limitation of fixture-based IST: the load you test against is not the load you will run. A load bank exists to consume power and reject heat on command. A GPU cluster running a synchronous training step does something a load bank was never built to do — it swings its entire draw, in lockstep, across thousands of accelerators, on the cadence of the collective-communication pattern. When the all-reduce stalls compute, tens of megawatts can fall in milliseconds; when compute resumes, it returns just as fast. This is the transient physics made canonical in Chapter 4.5, and it is the reason the GB300 NVL72 adds ~65 J/GPU of in-shelf energy storage and facility BESS designs add their own power-smoothing role. In July 2025, NVIDIA reported up to ~30% lower peak power on its instrumented GB200/Megatron demonstration of these features. IST poses it directly: which load-bank technology do you commission against, and therefore which of these dynamics do you leave un-demonstrated?

The table compares independent capabilities, not a ladder from resistors to perfect GPUs; price the missing waveform or heat path instead of buying a technology label. Resistive load banks are cheap, ubiquitous, and prove the steady-state power and cooling capacity — they are the right tool for the bulk of L4 and the soak portion of IST. Reactive banks add power-factor and ripple stress that resistive banks miss, exercising the UPS, the generator AVR, and protection devices closer to real conditions. Dynamic / AI-emulating banks — transistor-based or, for 800 VDC architectures, programmable DC loads — are a programmable option for the required step-load profile; their measured bandwidth, like that of switched resistive and reactive banks, decides which edges they can reproduce; they let you fire a scripted swing at the BBU and BESS mitigation stack and watch it absorb. Air-cooled electrical load banks leave the technology-cooling system unexercised. Liquid-cooled load banks and product-representative emulator racks validate the thermal-hydraulic loop before production GPUs arrive; a staged workload then supplies hardware- and workload-specific normal-operation evidence.

Modern AI facilities defend against GPU transients with a layered stack, and IST is where you decide how much of it you exercise on the real building versus accept on vendor witness test. The layers, from grid inward: the facility-level BESS (and any synchronous condenser) absorbing multi-megawatt swings and supporting ride-through (Chapter 4.5, Chapter 13.4); the BBU / UPS layer bridging power transfers; the in-rack / in-shelf capacitance (GB300's ~65 J/GPU, Vera Rubin's larger reservoir) catching the fastest edges; and the workload-side power smoothing — ramp-rate limiting and power capping via SMI/Redfish — that shaves the peak before it reaches the wire. The problem is recursive: a static load bank cannot create the transient the stack exists to absorb, so an IST run holding resistive or reactive banks flat proves only that operating point. Switched or programmable fixtures prove the stack's response inside their measured envelope; real workloads add the tested software-synchronized profile, never every possible future swing. That leaves an explicit risk allocation: which mitigation layers you require to be demonstrated absorbing a real swing during IST, and which you sign off by analysis and recharge-test, knowing the first true exercise comes during the proxy run.

Not every failure mode in Appendix F can or should be physically tripped on the real building. Some are too destructive (you do not deliberately rupture a coolant line on a live hall to watch the leak-detection-and-isolation sequence), some are impractical (you cannot induce a real utility-side 230 kV fault on demand), and some are covered acceptably by a witnessed factory or vendor test plus an installed-and-healthy check. IST forces an explicit triage of the FMEA catalog into four buckets, and the allocation is a risk decision the owner signs, not the Cx agent.

The table's second column needs an actual coverage record. Thermal ride-through, cooling failover and the alarm hierarchy can be tested against commanded disturbances on liquid banks and injected signals; residual risk depends on the missing branch, waveform or control interface, not a universal 'medium' rating. That is the structural consequence of the realism gap: the failure modes most specific to AI factories are precisely the ones a load bank can demonstrate against a fault but not against the workload. Each required criterion still receives PASS, FAIL or HOLD; the release bridge identifies which subsequent test closes each HOLD.

A defensible IST acceptance package does three things the legacy IT version never had to. First, it states what was proven and against what load — every passed scenario annotated with the load-bank class used, so the residual-risk register is honest about which results carry the realism caveat. Second, it defines the instrumented thresholds that gate GPU energization: coolant temperature and flow stability under the worst observed transition, critical-bus voltage and timing within the named terminal limits, CDU re-energization time inside the throttle window, and a clean, de-duplicated alarm record. Third, it names the deficiencies knowingly carried into the proxy run — the specific thermal-hydraulic dynamics and workload-synchronized electrical behavior outside the fixtures' measured coverage — so the proxy run is explicitly chartered to close them rather than being treated as a victory lap.

• Gate to energize GPUs: the utility-loss chain, transfers and design-basis fault set have passing evidence from approved methods at the stated boundary; thermal ride-through measured inside the coolant envelope; critical cooling confirmed on protected bus with re-energization time bounded; BMS/DCIM/SCADA agreement and a clean alarm hierarchy demonstrated.
• Carried to the proxy run (Chapter 13.9): CDU and secondary-loop control response to real transient heat flux; the cross-rack-synchronized power swing against the full mitigation stack; worst-case-branch thermal-hydraulics under realistic, not air-rejected, load.
• Carried to staged ramp (Chapter 13.10): validation of power-smoothing and ramp-rate-limit configuration against the actual model, preserving live-block redundancy as load builds.

The acceptance gate is therefore not 'IST passed' but 'IST passed, residual risk named, and the conditions for safely meeting the real workload are written down.' That handoff — from a building proven against faults to a building about to meet its first real dynamics — is the seam IST exists to make safe.

Buy the tests that close the required source, branch and control states, then release only their measured envelope. A bigger load-bank invoice does not close an unconnected liquid branch, and a successful GPU run does not clear an untested protective action. Choosing by that evidence lets the staged workload begin with a finite residual list instead of the building's first experiment.