Disaster Recovery, Business Continuity & Geographic Failover

Most of this Part has been about keeping a single facility running: redundancy topologies (Chapter 12.1), the goodput-versus-availability rethink (Chapter 12.2), the thermal and electrical paths that fail inside the fence. This chapter is about the failure that the fence cannot stop — the one that takes the whole site, the whole region, or the whole control plane at once. A transformer fire, a substation lost to weather, a fiber cut that isolates a campus, a cloud region's metadata anchor going dark, a wildfire-evacuation order, a ransomware event in the orchestration layer. When that happens, the question is no longer "is my UPS healthy?" It is: where does the work go, how fast, and did I pay to have somewhere for it to go?

The continuity decisions geography forces run in sequence: RTO and RPO targets set per workload class and priced by tier; active-active versus active-passive across regions, which fixes fleet count and cost; the capacity constraint that failover capacity is power-bound wherever activation cannot finish within the recovery deadline; the single dependencies (control plane, DNS, weights store, key personnel, single-vendor loops) that turn a local fault into an estate-wide one; the contractual DR obligations you owe tenants versus what the topology actually delivers; and the drills, runbooks, and FMEA tie-in that prove the plan works before you need it.

Two numbers govern every continuity decision. RTO (Recovery Time Objective) is how long you can be down before recovery — the wall-clock from failure to service restored. RPO (Recovery Point Objective) is how much work you can afford to lose — the gap back to the last durable state. They are independent: you can want fast recovery (low RTO) while tolerating some lost work (looser RPO), or the reverse. The cost of each tier rises steeply and non-linearly as either target approaches zero, which is why the first act of continuity engineering is to refuse a single estate-wide target and instead tier the workloads.

AI workloads split the RTO/RPO table more sharply than any traditional enterprise estate, because the archetypes have opposite tolerances (the same split that drove the redundancy logic in Chapter 1.2 and Chapter 1.3). Interactive inference is the revenue surface and the tightest tier: a user is waiting, an SLA is running, and a regional loss that takes minutes to absorb is a visible outage. AWS's Platinum resilience example uses a 15-minute RTO and 5-minute RPO for mission-critical workloads; set project targets per workload and recovery architecture. Training continuity is checkpoint-bounded, but RPO is not the snapshot-start cadence. At failure, select the newest valid checkpoint whose durable copy and dependencies survive the event; lost work reaches back to its captured progress, not its commit-completion time; capture time, background drain, commit completion, replication scope, and overlapping checkpoints all affect that boundary. RTO is separately bounded by state discovery, capacity reacquisition, restore, validation, and resume. Batch inference can queue and retry when its delivery deadline permits; deadline-bound batch needs capacity that finishes the remaining work on time. Spending the strongest DR posture on a batch pipeline is the same anti-pattern as commissioning duplicated, fault-tolerant power when checkpoint recovery already meets the service objective and maintenance or a single component fault may interrupt load.

Zero RPO and its cost have no universal percentage. Declare the protected state and disaster first, then measure capture, blocking snapshot, background drain, durable commit, restore and contention under the reference workload. Select the newest valid checkpoint that has committed and survives the event; its captured progress determines lost work. Chapter 9.4 owns the checkpoint timing boundaries. The worked example below shows why a recent commit can still contain state older than the RPO target.

The newest valid surviving capture is S. Its state age is 09:10 − 09:00 = 10 minutes; the two minutes since remote commit are not the lost-work age. Reject the five-minute RPO claim and acquire a newer recoverable state or change the recovery objective. Flip: at the same failure time, a valid surviving capture at or after 09:05 satisfies the age screen, provided commit and recovery dependencies also survive. Retain protected older versions for corruption recovery because replication can copy deletion or corruption. PyTorch distinguishes staging and upload completion; the choice here applies the captured-state boundary from Chapter 9.4.

What makes AI-factory DR different from a classic continuity playbook is that failover capacity cannot be conjured on demand: a tight recovery deadline turns compatible capacity into an advance procurement decision. In a traditional cloud estate, failover capacity is fungible and on-demand: a region fails, you spin up instances elsewhere, you pay the burst rate, you move on. In a power-bound AI estate, that escape hatch is closed. You cannot rent 200 MW of GPUs in a neighbouring region on the morning your primary burns, because that capacity does not exist as slack — every megawatt is contended, interconnection queues run years (the queue framing from Chapter 3.1), and a high-demand part like a current-generation accelerator is allocation-gated, not catalogue-stocked. Capacity must already be energized and racked when energization, delivery or activation would otherwise exceed the deadline. A reservation is useful only with a tested activation path.

That collapses the comforting cloud-era distinction between "reserve capacity" and "pay for it later." For the inference surface, the spare region is a real second factory, pre-paid, drawing real power, depreciating on the same contested 2–3-year bear-case economic clock as the primary (the depreciation reality of Chapter 1.3 and the economics chapter it points to). The continuity decision is therefore a capital-allocation decision, not an availability one: how many megawatts of idle-until-needed capacity will you underwrite, and can you make them earn while they wait?

The most important mitigation is to stop treating the spare as idle. Reverse-arbitrage the standby: run interruptible, RPO-loose work — batch inference, evaluation sweeps, synthetic-data generation, low-priority fine-tunes — on the failover fleet during normal operation, and pre-empt it within the tested recovery budget when the primary fails. This is the continuity analogue of the curtailable-load fast lane in power procurement: the spare region pays part of its own carry by doing displaceable work, and the failover event becomes a scheduler pre-emption rather than a cold start. The design constraint is that the pre-emptible workload must drain fast enough to hit the inference RTO — which is an orchestration property, not a hardware one.

The lower rows of the table rest on an assumption that fails exactly when it is needed. Pilot-light and cold standby look attractive because their steady cost is low — but their RTO assumes you can acquire the missing capacity when the disaster hits, and in a power-bound market that acquisition is what falls through. A cold-standby plan that depends on renting GPUs from a neocloud during a regional outage is a plan that competes for scarce capacity with every other operator whose primary just failed in the same correlated event (the wildfire, the heatwave-driven grid event, the regional storm). Cold standby is defensible for a workload that can wait through its measured acquisition and restore path — days, if that is what the path takes. An interactive endpoint with a shorter deadline needs active-active, hot standby or another prepared path that restores service in time; a long-latency batch service can buy less readiness.

For r regions, GPUs per region = ceil[18,000 / ((r−1) × 100 × 0.8)]. Two regions require 225 GPUs each, 450 total. Three require 113 each, 339 total. The one-region capacity equivalent is 225 GPUs; the ideal unrounded reserve factors are r/(r−1), or 2 and 1.5. Rounding makes the three-region factor 339/225 = about 1.5. These are capacity factors, not total-cost factors.

With three regions, normal demand is 6,000 requests/s each. After one loss, each survivor takes 9,000: 9,000/(113 × 100) = 79.6%, below the assumed 80% limit. Combined qualified capacity is 18,080 requests/s. Capacity PASS at the stated peak; a larger displaced peak or a slower destination requires a new calculation.

Select three regions on the capacity screen: 339 installed GPUs instead of 450, saving 111 GPUs while the assumed nine-minute recovery fits the 15-minute objective. Keep the outcome on HOLD pending the qualified throughput and a drill with primary-region access denied. Flip: at 113 GPUs per surviving region, the throughput crossover is 18,000/(2×113×0.8), about 99.6 requests/s per GPU. Use the unrounded threshold for the screen; at an assumed 95 requests/s, each region needs ceil[18,000/(2×95×0.8)] = 119 GPUs, so the 113-GPU design fails. Alternatively, any additional delay above six minutes on the sequential drill path misses the RTO. AWS’s recovery-pattern guidance supplies the method boundary; Chapter 13.9 owns the recovery acceptance evidence.

Geographic failover protects against losing a place. It does nothing against losing a shared dependency — a component that, when it fails, fails everywhere at once and renders your second region useless because it depended on the same thing. Apply the blast-radius lens from Chapter 12.1 across regions: enumerate every system that is common to all sites and ask what happens when it is the thing that breaks.

A concrete warning is the AWS US-EAST-1 outage of October 19–20, 2025: a latent race in DynamoDB’s DNS-management automation produced an empty regional endpoint record. Dependent services lost connectivity and recovered on different clocks; existing EC2 instances stayed healthy while new launches failed. A multi-region data plane still depends on a single-region control plane for operations that require it. Two 2026 events add the physical-plant counterpart. Coinbase’s June 1 postmortem reports simultaneous chiller failures on May 7 in one AWS use1-az4 data hall, a thermal rack shutdown, and roughly eight hours of unavailable or degraded core services followed by further recovery: surviving zones did not automatically make the application available. Google’s July 25 report describes a July 15 3 ms utility voltage drop in europe-west4-a, a failed DRUPS transfer and a chiller controller that did not restart the pumps while redundant cooling was unavailable during construction. The hall reached 44 °C; the report summarizes 14 hours, 55 minutes of service interruption, with separate timelines for VMware Engine, Bare Metal Solution and NetApp Volumes. This is the shared-control and unavailable-reserve combination in Chapter 12.1. Keep thermal excursion, cooling restoration and service restoration on separate clocks; derive the selected liquid rack’s maintained-flow and stopped-flow response from OEM data and tests, not these air-hall timings. Your inference can span three regions, but a scheduler, discovery layer, secrets store or model-registry metadata in one place still bounds operations that need it. Test the recovery region with those primary dependencies unavailable before counting its free GPUs as usable failover capacity.

The dependencies worth enumerating for an AI estate are specific. The control / orchestration plane (scheduler, fleet-management, health-checking) — anchor it in a single region and you have re-created US-EAST-1. The model-weights store — if the failover region cannot serve the current weights because replication lagged, your spare fleet boots into a stale or empty model; weights replication must be versioned, immutable, and ahead of the failover need, not behind it. DNS and global load-balancing — the very mechanism you rely on to drain a failed region is itself a global dependency that must not share fate with it. Shared firmware and single-vendor loops — a common-cause defect in a CDU controller, a BMC firmware revision, or a single liquid-cooling vendor's pump logic can take every site running that revision down together (the thermal-path reliability concern from Chapter 12.2); this is the beta-factor / common-cause-failure problem that the quantitative model in Chapter 12.5 exists to size. And people — covered next, because the bus-factor on a novel liquid-cooled estate is realer than most plans admit.

Geographic failover answers "what if the site is gone." Business continuity answers the harder, slower questions: what if the people are gone, the supply chain is gone, or the operating environment changes for months rather than hours. These are the low-probability, high-consequence tails that drills rarely exercise and plans rarely fund — and that the 2020-2022 period taught a generation of operators to take seriously.

Key-personnel continuity is sharper for AI factories than for legacy halls precisely because the technology is new. The number of engineers who can safely intervene on a direct-to-chip liquid-cooled NVL72-class rack mid-fault, or who understand a specific cluster's NCCL-level failure signatures, is small — often a handful per site, sometimes one. That bus-factor is a continuity risk equal to any transformer. The mitigations are unglamorous and effective: documented runbooks that a competent on-call engineer can execute cold (not tribal knowledge), cross-training and rotation so no single person is the only path to recovery, vendor field-service contracts with guaranteed response windows for the loops you cannot self-service, and a deliberate refusal to let the on-call roster narrow to one name.

Supply continuity is the slow disaster. AI-factory recovery depends on parts that are themselves allocation-gated: HBM and advanced-packaging supply is allocated a generation ahead (Micron reported its 2026 HBM output sold out; the other suppliers publish allocation, not availability), CDUs and quick-disconnects are specialized, high-voltage transformers and switchgear carry multi-year lead times. A spare-parts strategy sized for a legacy hall — a few PSUs and fans on a shelf — does not cover a liquid-cooled estate where the failed part may be a long-lead transformer or an allocation-gated accelerator whose quoted replacement time exceeds the service restoration deadline. Continuity here means stocking the critical-spares list deliberately, holding vendor SLAs with teeth, and accepting that some failures are recovered by reconfiguration (shrinking the cluster, re-routing the fabric) rather than replacement. Pandemic / access-denial continuity generalizes the personnel question: can the site run lights-out for an extended period with no on-site staff, can remote hands and zero-touch provisioning carry the load, and is the runbook executable by people who cannot physically enter the building? The facilities that rode 2020 best were the ones already operating close to lights-out by design.

DR is an engineering posture and a set of promises in a contract, and the gap between the promise and the topology is where operators get hurt. A colocation or capacity provider's master service agreement carries continuity obligations — availability commitments, maintenance-window rules, sometimes explicit DR or geographic-redundancy clauses — and the customer's own SLA to their users sits on top. When a real correlated failure exceeds the topology, the service-credit ladder fires and, worse, the reputational and renewal damage compounds. The contract has to describe what the topology can actually deliver under a realistic failure environment, not under the optimistic independence assumption.

Three contractual failure patterns recur. First, promising an availability number the single-region topology cannot reach — committing to four or five nines on a facility whose control plane, weights store, or sole liquid-cooling vendor is a single point of failure; the availability promise breaks when that common-cause event’s frequency and complete restoration time exhaust the contracted window’s loss budget. Second, silent capacity assumptions — a DR clause that implies failover capacity exists without specifying that it is pre-energized and reserved, so that the obligation is technically met on paper and impossible in a power-bound outage. Third, mismatched RTO/RPO tiers — selling every workload the same continuity grade rather than tiering it, which either over-charges the customer for batch-grade work or under-protects their inference surface. The detailed structure of availability-versus-goodput SLAs, the penalty and service-credit ladders, and how goodput shortfalls are measured and attributed is the subject of Chapter 12.4; the productization of these commitments to customers is in Chapter 10.9 and the serving-side SLOs in Chapter 10.11. One rule connects them: never contract a continuity grade your drills have not proven.

A DR plan that has never been executed is a hypothesis. Regular, adversarial drilling converts it into a capability — the deliberate, scheduled failure of a region, a control-plane component, or a dependency, with the recovery measured against its RTO/RPO target and the gaps fixed before the real event. This is chaos-engineering discipline applied to the facility and cluster layer: you do not wait for the wildfire to discover that your weights-replication lag exceeds your inference RPO, or that the failover load-balancer shares fate with the region it is supposed to drain.

The artifacts that make drills repeatable are runbooks — step-by-step recovery procedures, written to be executed cold by a competent on-call engineer who was not in the room when the architecture was designed. A good runbook names the trigger, the decision authority (who declares a failover, since premature failover during a transient is its own outage), the exact mechanical steps, the verification that the failover succeeded, and the fail-back procedure once the primary returns — because fail-back, the step everyone forgets to rehearse, is frequently where the second outage happens. Each runbook should map to a specific failure mode in the consolidated FMEA catalog in Appendix F: every mode the FMEA enumerates as estate-significant should have a corresponding rehearsed recovery, and every drill should exercise a mode the FMEA flagged. The emergency-operating-procedures that tie operations to that same catalog live in the commissioning and operations parts; the documentation and acceptance-test discipline that captures the go-live baseline against which recovery is measured is in Chapter 13.2.

Choose the least costly posture whose surviving state, compatible capacity and complete drill path meet the workload’s RPO and RTO. A cheaper standby that misses either target buys a recovery promise the service cannot use; more live capacity is justified only when it removes the deadline or loss constraint. Map physical initiators to Appendix F and adversarial initiators to Chapter 11.1, then exercise the same recovery authority and evidence.