DCIM, Telemetry & Observability for GPU-Dense, Liquid-Cooled Facilities

A GPU-dense, liquid-cooled AI factory is the most heavily instrumented building most operators will ever run, and also the one where the instrumentation most often fails to answer the only question that matters: why did goodput just drop? The reason is structural. Two telemetry universes coexist in the same hall and almost never share a clock, a namespace, or an on-call rotation. On one side is the IT plane — per-GPU temperature, power, SM clocks, ECC and XID/SXID error events, NVLink and fabric counters — sampled at one-to-ten-second cadence by agents like NVIDIA's DCGM exporter and scraped into a time-series database. On the other is the OT plane — CDU flow and pressure, coolant supply/return temperature, dew point, valve positions, pump speeds, leak-detection zones, switchgear and UPS state — polled at thirty-to-sixty-second cadence over Modbus, BACnet, SNMP, and increasingly Redfish, surfaced through a BMS/SCADA front end and a DCIM platform.

This chapter is about the decisions that determine whether those two universes become one observable system or stay two. We start from the stack itself — what each layer measures and at what rate — then take the central fork: siloed telemetry versus IT/facility correlation, and the protocol and data-architecture choices that make correlation possible or impossible. We treat liquid-cooling observability as its own discipline, because liquid is the subsystem where a missed signal turns into a flooded electrical room. We separate alarm management (an OT engineering discipline on the ISA-18.2 / EEMUA-191 lineage) from alerting (an SRE discipline of SLO burn), because conflating them is how you build a system nobody trusts. And we close on the operational twin — the live, as-built sibling of the design-validation twin from Chapter 2.7 — and the autonomy ladder a DCIM climbs from dashboard to closed-loop control.

The instinct of a newcomer is to treat telemetry as a volume problem — collect everything, store it forever, and the answers will be in there somewhere. In a 100,000-GPU campus that instinct produces a monitoring system that is simultaneously too expensive and unable to answer an incident. The right framing is a layered stack, where each layer has a native protocol, a native cadence, and a native owner, and the engineering question is how the layers connect, not how much each one emits.

The IT/AI telemetry layer is the fast one. Per-GPU health flows from the device through NVML into DCGM and out via an exporter in Prometheus line format; XID and SXID events (the GPU and NVSwitch error codes) are the load-bearing failure signals, and on a healthy device the error counters — remapped rows, PCIe replay, uncorrectable ECC — sit at zero, so a nonzero value is itself the alarm. Above the device sit the fabric counters (InfiniBand/UFM or Ethernet/NetQ), the scheduler's view of job and node state, and the application's own loss curve and step time. This layer's cadence is seconds, its cardinality is very high — a single GB200 NVL72 rack is 72 GPUs times dozens of metrics each — and its consumer is the SRE who needs to cordon a lemon node before it poisons a collective.

The facility/OT layer is the slow, durable one. Power chain (switchgear, UPS, PDU, branch-circuit metering), cooling plant (chillers, dry coolers, CDUs, pumps, valves), environmental sensors, and physical-security systems report through PLCs and a BMS into a SCADA/DCIM front end. Its cadence is tens of seconds to minutes, its cardinality is modest, and its consumer is the facility engineer and the alarm system. Critically, this layer is governed by a different culture: it values determinism, alarm rationalization, and long-horizon trending over the high-cardinality, ephemeral-by-design ethos of the IT side. A thermal event originates in the OT layer and manifests as a goodput collapse in the IT layer, and the two layers, left to themselves, will not tell you they are the same event.

Correlation is not a dashboard feature you switch on; it is a data-architecture decision made at the seam between the layers above. Three sub-decisions determine whether the seam holds.

The protocol convergence decision. Historically the IT plane spoke IPMI and the OT plane spoke Modbus/BACnet, and never the twain met. The 2026 lever is Redfish (the DMTF's RESTful successor to IPMI), which has crossed from a server-management API into a facility-telemetry API: the DMTF added liquid-cooling equipment to Redfish in the 2023.1 release (the DSP2064 model for CDUs, coolant loops, leak detection, pumps, and reservoirs), so a CDU and a GPU can now expose their state through the same schema and HTTP transport. That does not eliminate the BMS — chillers, switchgear, and legacy plant still speak BACnet and Modbus — but it gives you a convergence target. The fork is whether you adopt Redfish-as-the-common-model and bridge the stragglers, or keep a polyglot architecture and pay for translation at every integration. Above the wire, OpenTelemetry is becoming the lingua franca for the IT side's metrics, traces, and logs, which lets application step-time and node health share a schema with the SRE tooling.

The time-sync and namespace decision. Correlation is impossible without a shared clock and a shared identity. A throttle event timestamped by a GPU's monotonic clock and a flow anomaly timestamped by a PLC's local clock cannot be aligned to causality if the clocks drift by seconds — and at one-second IT cadence, seconds of skew destroy the signal. PTP/NTP discipline across both planes is a precondition, not a nicety. Equally, every datum needs an asset key that resolves the physical topology: GPU → node → rack → CDU → branch circuit → PDU → room. Without that graph you cannot answer "which GPUs are downstream of the pump that just slowed," which is the entire value proposition of correlation.

The store-and-retain decision. A GPU-dense fleet is a cardinality bomb. Store every per-GPU metric at full resolution forever and your time-series database cost rivals a non-trivial slice of the power bill; store too little and the one transient you needed is gone before the postmortem. The mature pattern is tiered: high-resolution hot storage for days, downsampled rollups for months, and event-triggered full-fidelity capture (a "flight recorder" that keeps raw samples around any XID, throttle, or leak event). The decision you are actually making is where to put the blind spot — and the only defensible answer is to keep full fidelity exactly where incidents originate and accept lossy aggregation everywhere else.

In an air-cooled hall, a cooling failure is a slow thermal ramp that buys you minutes and at worst trips a thermal limit. In a direct-to-chip liquid hall, the failure modes are faster and more consequential, and the telemetry must be designed around them specifically. There are three you instrument for above all others.

Throttle-by-deviation. The GB200 NVL72 envelope is tight — coolant inlet below ~25 °C, roughly 80 L/min flow, under ~10 °C rise across the cold plates — and a deviation does not fail the rack, it throttles the GPUs up to ~50%. That signal is invisible to a naive availability monitor: every node is "up," every GPU is "healthy," and goodput has silently halved. Only by correlating the CDU's flow/temperature telemetry with the GPUs' clock-throttle reason codes do you see that a cooling deviation, not a software bug, is eating the run. This is the single most important correlation in a liquid-cooled facility.

Leak detection. Liquid above an electrical room is a step-change in risk, so leak telemetry is zoned, redundant, and wired to act: rope/spot sensors per rack and per CDU, pressure-decay detection on the loops, and reservoir-level trending that catches slow loss before it becomes a puddle. The mature posture pairs detection with negative-pressure loop design and automated isolation valves, so a confirmed leak can drain rather than spray. The telemetry decision is the confidence threshold: a false trip drops a rack of GPUs, a missed trip wets a busbar, and the gate is set so an isolation command fires only when leak sensors agree and flow/pressure confirm.

Fluid quality and slow drift. The signals that predict a failure weeks out are not dramatic: conductivity and pH creep, particulate and filter-differential pressure, biological fouling, dew-point margin narrowing toward condensation. These feed the predictive-maintenance program (canonical in Chapter 14.5) rather than the alarm system, and they are exactly the trends a siloed IT stack never sees and a correlated plane surfaces as a degrading-CDU early warning.

The most common cultural failure in an AI-factory operations center is treating the OT alarm system and the IT alert system as the same thing. They are different disciplines with different standards, and merging them naively produces the worst of both: an alarm flood nobody can action and an SLO-burn signal nobody trusts.

Alarm management is an OT engineering discipline on the ISA-18.2 / IEC 62682 / EEMUA-191 lineage, and its core insight is that an alarm is a request for operator action, not a notification. Every alarm must survive rationalization: it is justified against a written alarm philosophy, documented with a consequence, a required operator action, and a time-to-respond, and anything that fails that test is not an alarm. The discipline exists because of a quantifiable failure mode — the alarm flood, conventionally defined as more than ten new alarms in ten minutes, the rate above which a human operator is overwhelmed and starts ignoring the board. The standard's countermeasures are specific: rationalization to cut nuisance alarms, prioritization so the critical ones stand out, shelving (the operator-controlled, time-bounded, audited suppression of a known nuisance alarm), and state-based suppression so a planned maintenance evolution does not generate a thousand expected alarms. In a liquid-cooled hall this matters acutely: a single CDU trip can cascade flow, temperature, and throttle alarms across an entire row, and without designed flood suppression the operator sees a wall of red and cannot find the root cause inside it.

Alerting is the SRE discipline of paging on SLO burn — you define goodput, latency, or error budgets and alert when the rate of consumption threatens the objective, deliberately avoiding alerts on every transient symptom. The two systems must coexist, and the operator's decision is how they relate. The wrong answer is to dump every OT alarm into the IT pager or every IT symptom onto the OT board. The right answer is a federated model: each system rationalized to its own discipline, with a deliberate, narrow set of cross-links — a confirmed cooling alarm that is eating goodput should reach the training SRE, and a fleet-wide goodput collapse should prompt the facility operator to look at the plant. Correlation feeds both; it does not merge them.

A DCIM is not one thing; it is a position on a ladder, and the operator's strategic decision is how far up that ladder to climb — because each rung trades a real capability against a real risk. The rungs are: read-only visualization (dashboards and trends, no analytics); advisory analytics (anomaly detection and predictive-maintenance lead times, human acts); closed-loop control (the DCIM moves setpoints — pump speed, valve position, power caps — within bounded authority); and the operational digital twin (a live, as-built model that simulates the consequence of an action before committing it, and increasingly runs an agentic control loop). The 2026 reference designs — NVIDIA's Omniverse DSX digital-twin blueprint, the hyperscalers' in-house plant analytics — point at the top of the ladder, but most operators are correct to climb deliberately rather than leap.

The crucial distinction this chapter draws is between the operational twin and the design-validation twin of Chapter 2.7. The design twin is built before steel is cut to validate a thermal and power design against a model; it is a planning artifact. The operational twin is the design twin's live descendant, continuously reconciled against as-built reality — real sensor streams, real failed pumps, real firmware revisions — so that when it predicts the effect of raising a setpoint, it is predicting against the building that exists, not the one that was drawn. The decision that makes or breaks an operational twin is the as-built fidelity loop: a twin that drifts from reality (a swapped CDU, a re-cabled rack, a derated chiller the model does not know about) gives confidently wrong predictions, which is worse than no twin at all. Closing that loop — continuous and re-commissioning telemetry feeding the model — is the subject of Chapter 14.14, and the autonomy decisions the twin enables are taken up in Chapter 14.13.

The final discipline is editorial: a GPU-dense facility can emit more telemetry than anyone will ever query, so the question is not what you can measure but what is load-bearing for the four numbers that govern the business — goodput, PUE, time-to-detect, and time-to-cordon. Work backward from each.

• Goodput (the metric canonicalized in Chapter 14.1) is protected by the correlation of throttle-reason codes, XID/SXID events, fabric link flaps, and CDU deviation — the signals that explain why effective training time dropped. If a metric does not help you attribute a goodput loss, it is not load-bearing for goodput.
• PUE / WUE is protected by the power-chain and heat-rejection metering — branch-circuit power, CDU and chiller energy, water draw — trended over seasons. This is the OT plane's classic job and the place its long-horizon retention earns its keep.
• Time-to-detect is set by the fastest plane's cadence and the quality of the failure signals on it — which is why per-GPU XID monitoring at one-second cadence, not minute-scale facility polling, is what catches a lemon node before it poisons a collective.
• Time-to-cordon is set by how directly detection wires to action — the cordon-and-drain path that ejects a failing node, fed by the same health telemetry, treated in Chapter 10.7 (autonomous recovery) and the failure taxonomy of Chapter 14.3.

Everything else is, at best, useful context and, at worst, vanity telemetry you are paying to store. The cardinality-and-retention bargain from earlier in the chapter is the same decision seen from the cost side: keep full fidelity where it serves one of these four numbers, downsample everywhere else, and resist the impulse to retain everything forever because storage felt cheap on day one and did not stay that way at fleet scale.