Commissioning Fundamentals, Levels & Program Governance

Commissioning (Cx) is the discipline that refuses to take the design's word for it. Every prior part of this guide makes claims — that the power chain will ride through a utility dip, that the cold plates will hold the GPUs under sustained load, that the fabric will carry a non-blocking all-reduce, that the standby plant will pick up the building before the UPS runs flat. Commissioning is the structured process of generating the evidence that those claims are true under the conditions that matter, before the facility carries revenue load. It is the bridge between Part 2's design and Part 14's day-2 operations, and it is the last point in the project where a latent integration defect costs hours to fix rather than a multi-day outage on a live cluster.

For a conventional enterprise data center this is a mature, well-codified ritual. For an AI factory it is not — and the gap is the subject of this entire Part. The reason is structural: an AI facility is two machines pretending to be one. There is the facility — substation, switchgear, generators, UPS/BESS, CDUs, chillers, piping, BMS — which the commissioning industry knows how to accept. And there is the cluster — tens of thousands of GPUs, the scale-up and scale-out fabric, storage, and the scheduler — which the facility commissioning standards barely acknowledge exists. The two tracks run in parallel, share critical resources (power and cooling), and must be interlocked at specific gates, or each track signs off in isolation against a load the other half cannot actually deliver. This chapter establishes the fundamentals — the levels, the two tracks, the roles, and the governing documents — that the rest of Part 13 builds on.

Mission-critical commissioning is organized as a ladder of increasing integration, conventionally Level 1 through Level 5 (with some programs adding a Level 0 design-review and a Level 6 post-occupancy / seasonal stage). Each rung tests a wider boundary than the last; you do not climb to the next rung until the current one is signed and its deficiencies closed. The discipline of the ladder is that integration faults are found at the lowest level at which they can possibly appear — a mis-wired CT caught at L3 standalone test is a morning's rework; the same fault discovered at L5 during a black-building test invalidates the run and resets days of scripted work.

The names below are the most common form; vendors and owners vary the labels, but the boundary each rung tests is stable. The two parallel tracks (next section) each have their own analogue of this ladder, and Chapter 13.2 turns each rung into a quantitative script with explicit pass/fail gates.

Two things about the ladder are easy to get wrong and expensive to recover from. First, L4 already includes failure-mode testing within a discipline — a UPS must demonstrate its own transfer and its battery ride-through at L4, not wait for L5. L5 then tests failure modes across disciplines, where the interesting cascades live (a generator that starts fine and a cooling plant that rides through, but a BMS sequence that does not hand the load off in the right order). Second, the temptation under schedule pressure is to truncate the ladder — to skip a real L5 and call functional testing 'good enough.' That truncation pushes the integration risk onto day two.

The L5 test matrix is dictated by the resilience claim the facility was designed to meet, and the two claims that matter are the Uptime Institute's concurrent maintainability (Tier III) and fault tolerance (Tier IV). They sound similar and they are not, and commissioning is where the difference becomes physical.

Concurrent maintainability (Tier III) means any single capacity component or distribution path can be removed from service — for planned maintenance — with no impact on the IT load. The IST must therefore prove that you can take down each path in turn, one at a time, deliberately, and the load never notices. It does not require surviving an unplanned simultaneous fault. Fault tolerance (Tier IV) is the stronger claim: the facility survives any single unplanned worst-case failure — including the loss of an entire distribution path to fire or flood — with no load impact, while still being concurrently maintainable. Tier IV adds compartmentalization and continuous cooling requirements that Tier III does not, and the IST must demonstrate the auto-response to a fault, not just an orderly manual switchover.

The consequence for the commissioning program is direct: the resilience target you committed to in design writes your failure-mode demonstration script. A Tier III program scripts orderly, one-path-at-a-time maintenance scenarios. A Tier IV program scripts unannounced faults, dual-path compartment loss, and continuous-cooling ride-through — a materially larger, riskier, longer IST. Choosing the tier is a design and capital decision (Chapter 12.1); but it is at commissioning that you discover whether the building actually earns it, and that is not a decision to defer to the IST.

Here is the structural fact that makes AI-factory commissioning different from every commissioning program that came before it. There are two acceptance ladders running at once, on different schedules, owned by different organizations, governed by different standards — and they are coupled by shared physical resources.

The facility track (Chapters 13.3–13.6) is the classical mission-critical Cx program: utility energization, switchgear, generators, UPS/BESS, CDUs, chillers, piping, controls, culminating in the L5 IST. It is owned by the owner's CxA and the MEP trades, governed by ASHRAE Guideline 0/1.1/1.6, Uptime, and BICSI 002. The cluster track (Chapters 13.7–13.9) is the IT validation program: fabric commissioning (BER, link-flap, topology, bandwidth/latency), GPU node burn-in and diagnostics, cluster-scale NCCL/collective acceptance, and a reference/proxy training run. It is owned by the platform/SRE/ML-infra team, governed by vendor deployment guides and de-facto standards like SemiAnalysis ClusterMAX — none of which the facility CxA typically touches.

If those two tracks each sign off in isolation, you have certified nothing useful. The facility track accepts a power-and-cooling envelope using load banks — resistive heaters that reject to air and draw a smooth, steady load. The cluster track accepts compute using real GPUs that draw a violent, synchronized, switching load and reject heat into liquid. Neither track, alone, exercises the seam between them. The interlock gates are where that seam is tested.

The middle row is the defining limitation of facility commissioning for AI and the reason this Part exists. A load bank is a deliberately boring load: a bank of resistive elements that draws a smooth, constant current and rejects its heat straight to the room air. A frontier GPU cluster is the opposite — tens of thousands of accelerators that ramp from idle to full and back in milliseconds in lockstep across a synchronous step, imposing power transients and harmonic content that a resistive bank never produces, and dumping that heat into a liquid loop the load bank never touches. The facility track can prove the power chain is wired correctly and redundant; it cannot prove the chain absorbs the dynamic load swing of a real all-reduce, because the tool it uses to emulate load cannot generate that swing. This is the dynamic-load realism gap, and it is canonicalized in Chapter 13.6; the transient physics behind it lives in Chapter 4.5. The mitigation is sequencing: the proxy training run of Chapter 13.9 is the first and only acceptance test that closes the gap, which is why it must come before — not after — go-live.

Commissioning is only as good as the requirements it tests against, and those requirements live in a short chain of governing documents that every acceptance script must trace back to. The chain is deliberately a chain — each document derives from the one before it, so that a pass/fail gate in an L5 script can be followed all the way back to a stated owner intent.

• Owner's Project Requirements (OPR). The owner's intent in measurable terms: the availability target (and therefore the Tier or BICSI class), the density and ramp the building must accommodate, the environmental envelopes, the maintainability expectations, and the acceptance criteria the owner will hold the project to. Everything downstream is an answer to the OPR. For an AI factory the OPR must state the workload intent (training-shaped vs inference-shaped, per Chapter 1.1), because that is what makes the difference between commissioning for goodput and commissioning for availability.
• Basis of Design (BOD). The design engineer's documented explanation of how the proposed systems satisfy each OPR requirement — the topology, the redundancy scheme, the cooling architecture, the setpoints, and the assumptions. ASHRAE Guideline 1.1 governs the BOD specifically. The CxA verifies the BOD answers the OPR before construction, not after.
• Sequence of Operations (SOO). The control-logic specification — exactly how the BMS/EPMS/DCIM is supposed to behave in every normal, maintenance, and failure mode. The SOO is the script-writer's bible: an L4/L5 functional test is, in essence, a line-by-line proof that the real controls match the SOO under real conditions. A vague or incomplete SOO is the most common root cause of a failed IST, because there is nothing precise to test against.

The role that owns this chain is the Commissioning Authority (CxA) — and the single most important governance decision is to engage the CxA at the design phase, as an independent party, not as a general-contractor self-check bolted on at the end. ASHRAE Guideline 0 (the process), ASHRAE Standard 202 (the formalized commissioning process / Cx-Process), and the data-center-specific ASHRAE Guideline 1.6 all assume an independent CxA verifying the OPR→BOD→SOO chain across the whole project lifecycle, from design review (L0) through post-occupancy. An owner who engages the CxA only to 'witness the IST' has already lost the design-phase reviews where the cheapest defects are caught.

Pull the threads together and the shape of the problem is clear. A conventional data-center commissioning program is a single, well-standardized ladder culminating in an IST against load banks, governed end-to-end by ASHRAE/Uptime/BICSI, signed by an independent CxA. That program, applied unchanged to an AI factory, certifies a building that has never seen its actual load and a cluster the standards never mention.

The three structural breaks are: (1) two tracks, not one — facility and cluster, on different schedules and standards, requiring explicit interlock gates; (2) load realism — the load banks that accept the facility cannot reproduce the dynamic power and liquid-side thermal behavior of real GPUs, so the proxy training run becomes a mandatory acceptance gate rather than an optional nicety; and (3) density ramp — the facility is commissioned today against a 130 kW/rack generation while the design must accommodate the 600 kW-class racks of the next one, so the IST must validate not only the steady state but the headroom the ramp will consume (floor, water, electrical, cooling-plant turndown). A program that handles these three is a Part 13 program. A program that ignores them is a conventional program that will hand a long deficiency list to day-2 operations — exactly the failure environment the 419-interruptions-in-54-days reality of a real training run cannot afford.