Disaster Recovery, Business Continuity & Geographic Failover

Most of this Part has been about keeping a single facility running: redundancy topologies (Chapter 12.1), the goodput-versus-availability rethink (Chapter 12.2), the thermal and electrical paths that fail inside the fence. This chapter is about the failure that the fence cannot stop — the one that takes the whole site, the whole region, or the whole control plane at once. A transformer fire, a substation lost to weather, a fiber cut that isolates a campus, a cloud region's metadata anchor going dark, a wildfire-evacuation order, a ransomware event in the orchestration layer. When that happens, the question is no longer "is my UPS healthy?" It is: where does the work go, how fast, and did I pay to have somewhere for it to go?

This chapter works through the continuity decisions geography forces. It defines RTO and RPO per workload class and prices each continuity tier; draws the master fork — active-active versus active-passive across regions — and traces what each does to fleet count and cost; confronts the 2026 reality that failover capacity is power-bound and must be energized in advance, not rented on the day; catalogues the single dependencies (control plane, DNS, weights store, key personnel, single-vendor loops) that turn a local fault into an estate-wide one; maps the contractual DR obligations you owe tenants against what the topology actually delivers; and closes on drills, runbooks, and the FMEA tie-in that prove the plan works before you need it.

Two numbers govern every continuity decision. RTO (Recovery Time Objective) is how long you can be down before recovery — the wall-clock from failure to service restored. RPO (Recovery Point Objective) is how much work you can afford to lose — the gap back to the last durable state. They are independent: you can want fast recovery (low RTO) while tolerating some lost work (looser RPO), or the reverse. The cost of each tier rises steeply and non-linearly as either target approaches zero, which is why the first act of continuity engineering is to refuse a single estate-wide target and instead tier the workloads.

AI workloads split the RTO/RPO table more sharply than any traditional enterprise estate, because the archetypes have opposite tolerances (the same split that drove the redundancy logic in Chapter 1.2 and Chapter 1.3). Interactive inference is the revenue surface and the tightest tier: a user is waiting, an SLA is running, and a regional loss that takes minutes to absorb is a visible outage. Practitioner targets cluster around a ~15-minute RTO and a ~5-minute RPO for production inference (Introl DR analysis, 2025). Training is the opposite: the job already checkpoints, so its RPO is simply the checkpoint interval (commonly 2-4 hours of replicated state), and its RTO is bounded by how fast you can re-acquire GPUs and resume from the last checkpoint — hours, not minutes, are tolerable. Batch inference is best-effort: it queues, it retries, it can wait for the primary to come back. Spending inference-grade DR on a batch pipeline is the same anti-pattern as commissioning 2N power for a checkpointable training job — buying a tier the workload does not value.

Read the cost column as the thing you are actually buying. Zero RPO is not free even when it sounds like good hygiene: forcing a training job to a zero-RPO posture (synchronous cross-region state) imposes a ~15-20% throughput penalty on the run (Introl, 2025) — you are paying in goodput for a continuity tier the checkpoint already provides for nearly free. The honest move is to set RPO equal to the checkpoint cadence and spend the saved bandwidth on more GPUs. The checkpoint math that makes this defensible — interval selection, multi-tier checkpointing, sub-2-minute restart — is canonical in Chapter 9.4; here it is simply the input that sets training's RPO floor.

What makes AI-factory DR different from every continuity playbook written before 2024 is that failover capacity can no longer be conjured on demand. In a traditional cloud estate, failover capacity is fungible and on-demand: a region fails, you spin up instances elsewhere, you pay the burst rate, you move on. In a power-bound AI estate, that escape hatch is closed. You cannot rent 200 MW of GPUs in a neighbouring region on the morning your primary burns, because that capacity does not exist as slack — every megawatt is contended, interconnection queues run years (the queue framing from Chapter 3.1), and a high-demand part like a current-generation accelerator is allocation-gated, not catalogue-stocked. The failover capacity must already be energized and the GPUs already racked before the disaster, or it is not failover — it is a wish.

That collapses the comforting cloud-era distinction between "reserve capacity" and "pay for it later." For the inference surface, the spare region is a real second factory, pre-paid, drawing real power, depreciating on the same 2-3 year economic clock as the primary (the depreciation reality of Chapter 1.3 and the economics chapter it points to). The continuity decision is therefore a capital-allocation decision disguised as an availability one: how many megawatts of idle-until-needed capacity will you underwrite, and can you make them earn while they wait?

The most important mitigation is to stop treating the spare as idle. Reverse-arbitrage the standby: run interruptible, RPO-loose work — batch inference, evaluation sweeps, synthetic-data generation, low-priority fine-tunes — on the failover fleet during normal operation, and pre-empt it instantly when the primary fails. This is the continuity analogue of the curtailable-load fast lane in power procurement: the spare region pays part of its own carry by doing displaceable work, and the failover event becomes a scheduler pre-emption rather than a cold start. The design constraint is that the pre-emptible workload must drain fast enough to hit the inference RTO — which is an orchestration property, not a hardware one.

The table hides a trap in its lower rows. Pilot-light and cold standby look attractive because their steady cost is low — but their RTO assumes you can acquire the missing capacity when the disaster hits, and in a power-bound market that assumption is exactly the one that fails. A cold-standby plan that depends on renting GPUs from a neocloud during a regional outage is a plan that competes for scarce capacity with every other operator whose primary just failed in the same correlated event (the wildfire, the heatwave-driven grid event, the regional storm). Cold standby is honest only for workloads that can genuinely wait days; for the inference surface, the realistic choices are active-active or hot standby, and both mean pre-energized megawatts.

Geographic failover protects against losing a place. It does nothing against losing a shared dependency — a component that, when it fails, fails everywhere at once and renders your second region useless because it depended on the same thing. The discipline here is the blast-radius lens from Chapter 12.1 applied across regions: enumerate every system that is common to all sites and ask what happens when it is the thing that breaks.

The canonical 2026 warning is the AWS US-EAST-1 outage of October 19-20, 2025: a latent race condition in DynamoDB's DNS-management automation produced an empty DNS record for a regional endpoint, and because so many global services anchor their metadata and control-plane operations in that one region, a ~15-hour event rippled worldwide — taking down services whose own architecture was nominally multi-region but whose control plane was not (AWS post-event summary; InfoQ; ThousandEyes, 2025). The lesson is precise: a multi-region data plane with a single-region control plane is a single-region system. Your inference can be replicated across three regions, but if the scheduler, the service-discovery layer, the secrets store, or the model-registry metadata lives in one place, that place is your true availability ceiling.

The dependencies worth enumerating for an AI estate are specific. The control / orchestration plane (scheduler, fleet-management, health-checking) — anchor it in a single region and you have re-created US-EAST-1. The model-weights store — if the failover region cannot serve the current weights because replication lagged, your spare fleet boots into a stale or empty model; weights replication must be versioned, immutable, and ahead of the failover need, not behind it. DNS and global load-balancing — the very mechanism you rely on to drain a failed region is itself a global dependency that must not share fate with it. Shared firmware and single-vendor loops — a common-cause defect in a CDU controller, a BMC firmware revision, or a single liquid-cooling vendor's pump logic can take every site running that revision down together (the thermal-path reliability concern from Chapter 12.2); this is the beta-factor / common-cause-failure problem that the quantitative model in Chapter 12.5 exists to size. And people — covered next, because the bus-factor on a novel liquid-cooled estate is realer than most plans admit.

Geographic failover answers "what if the site is gone." Business continuity answers the harder, slower questions: what if the people are gone, the supply chain is gone, or the operating environment changes for months rather than hours. These are the low-probability, high-consequence tails that drills rarely exercise and plans rarely fund — and that the 2020-2022 period taught a generation of operators to take seriously.

Key-personnel continuity is sharper for AI factories than for legacy halls precisely because the technology is new. The number of engineers who can safely intervene on a 130 kW direct-to-chip liquid-cooled rack mid-fault, or who understand a specific cluster's NCCL-level failure signatures, is small — often a handful per site, sometimes one. That bus-factor is a continuity risk equal to any transformer. The mitigations are unglamorous and effective: documented runbooks that a competent on-call engineer can execute cold (not tribal knowledge), cross-training and rotation so no single person is the only path to recovery, vendor field-service contracts with guaranteed response windows for the loops you cannot self-service, and a deliberate refusal to let the on-call roster narrow to one name.

Supply continuity is the slow disaster. AI-factory recovery depends on parts that are themselves allocation-gated: HBM and advanced-packaging supply is sold out generations ahead, CDUs and quick-disconnects are specialized, high-voltage transformers and switchgear carry multi-year lead times. A spare-parts strategy sized for a legacy hall — a few PSUs and fans on a shelf — does not cover a liquid-cooled estate where the failed part may be a 128-week-lead transformer or an allocation-gated accelerator. Continuity here means stocking the critical-spares list deliberately, holding vendor SLAs with teeth, and accepting that some failures are recovered by reconfiguration (shrinking the cluster, re-routing the fabric) rather than replacement. Pandemic / access-denial continuity generalizes the personnel question: can the site run lights-out for an extended period with no on-site staff, can remote hands and zero-touch provisioning carry the load, and is the runbook executable by people who cannot physically enter the building? The facilities that rode 2020 best were the ones already operating close to lights-out by design.

DR is not only an engineering posture; it is a set of promises in a contract, and the gap between the promise and the topology is where operators get hurt. A colocation or capacity provider's master service agreement carries continuity obligations — availability commitments, maintenance-window rules, sometimes explicit DR or geographic-redundancy clauses — and the customer's own SLA to their users sits on top. When a real correlated failure exceeds the topology, the service-credit ladder fires and, worse, the reputational and renewal damage compounds. The discipline is to make the contract describe what the topology can actually deliver under a realistic failure environment, not under the optimistic independence assumption.

Three contractual traps recur. First, promising an availability number the single-region topology cannot reach — committing to four or five nines on a facility whose control plane, weights store, or sole liquid-cooling vendor is a single point of failure; the number is unreachable the moment a common-cause event hits. Second, silent capacity assumptions — a DR clause that implies failover capacity exists without specifying that it is pre-energized and reserved, so that the obligation is technically met on paper and impossible in a power-bound outage. Third, mismatched RTO/RPO tiers — selling every workload the same continuity grade rather than tiering it, which either over-charges the customer for batch-grade work or under-protects their inference surface. The detailed structure of availability-versus-goodput SLAs, the penalty and service-credit ladders, and how goodput shortfalls are measured and attributed is the subject of Chapter 12.4; the productization of these commitments to customers is in Chapter 10.9 and the serving-side SLOs in Chapter 10.11. The single rule that connects them: never contract a continuity grade your drills have not proven.

A DR plan that has never been executed is a hypothesis. The only thing that converts it into a capability is regular, adversarial drilling — the deliberate, scheduled failure of a region, a control-plane component, or a dependency, with the recovery measured against its RTO/RPO target and the gaps fixed before the real event. This is chaos-engineering discipline applied to the facility and cluster layer: you do not wait for the wildfire to discover that your weights-replication lag exceeds your inference RPO, or that the failover load-balancer shares fate with the region it is supposed to drain.

The artifacts that make drills repeatable are runbooks — step-by-step recovery procedures, written to be executed cold by a competent on-call engineer who was not in the room when the architecture was designed. A good runbook names the trigger, the decision authority (who declares a failover, since premature failover during a transient is its own outage), the exact mechanical steps, the verification that the failover succeeded, and the fail-back procedure once the primary returns — because fail-back, the step everyone forgets to rehearse, is frequently where the second outage happens. Each runbook should map to a specific failure mode in the consolidated FMEA catalog in Appendix F: every mode the FMEA enumerates as estate-significant should have a corresponding rehearsed recovery, and every drill should exercise a mode the FMEA flagged. The emergency-operating-procedures that tie operations to that same catalog live in the commissioning and operations parts; the documentation and acceptance-test discipline that captures the go-live baseline against which recovery is measured is in Chapter 13.2.