Sizing, Data Gravity & Resilience

You now have the pieces: the parallel filesystem (Chapter 9.2), the NVMe data path (Chapter 9.3), the checkpoint math (Chapter 9.4), the loader (Chapter 9.5), the object tier (Chapter 9.6), and the inference KV hierarchy (Chapter 9.7). This chapter assembles them into a sized, sited, survivable subsystem. What makes the assembly hard is that storage failure is silent. A mis-sized fabric throws an error; a mis-sized storage tier starves the GPUs, and the loss surfaces as a utilization number a few points below where it should be, indistinguishable at a glance from a hundred other causes.

Four decisions structure what follows. Sizing: the storage:compute ratio and the bandwidth budget that decide whether the accelerators stay fed. Network co-design: how the checkpoint incast is kept off the training collectives, the placement that most affects storage goodput. Data gravity: whether you move the corpus to the compute or the compute to the corpus, and the egress economics that govern multi-site strategy. Resilience and TCO: durability, multi-tenancy QoS, security, and the benchmarking-and-acceptance regime that turns a vendor claim into a contractual commitment.

The question Chapter 9.1 opened decides everything downstream: is each tier bandwidth-sized or capacity-sized? A hot tier feeding packed sequential shards or absorbing a checkpoint drain is bandwidth-sized — the governing number is GB/s per GPU, not petabytes. One serving a directory of billions of small files is metadata-sized, and one serving KV fetches is latency-sized; run the personality test from Chapter 9.1 before you name the axis. Almost every AI capacity tier is capacity-sized — it holds the corpus and the checkpoint history, and the governing number is cost per TB. Size the hot tier by capacity and you buy a vast array that cannot saturate the GPUs; size the capacity tier by bandwidth and you overpay for flash speed the cold corpus never touches. Either error strands the asset you under-prioritized.

The hot-tier budget starts from a per-GPU read target. The reference architectures give you tiers, not one floor. NVIDIA's B300 Standard tier is 80 GB/s read per 256-GPU scalable unit — 0.31 GB/s/GPU — against ~0.98 for the Enhanced tier; the DGX H200 guidelines span 0.5, 1 and 5 GB/s/GPU for Good, Better and Best, with ~4 GB/s/GPU for computer vision, with other workloads requiring their own measured read and write budgets. NVIDIA says outright that those tiers assume a mix of workloads and that you should characterize your own first. So derive the read requirement from what the loader consumes — samples or tokens per GPU-second × stored bytes per sample, adjusted for cache hit rate and concurrent jobs — multiply by GPU count, and derive the write floor from serialized checkpoint bytes and the drain window you will accept. NVIDIA's named reference tier’s published read/write pair is a cross-check on the derivation, not a replacement for it; use the exact pair rather than rounding it into a universal half-read rule. This is why the SuperPOD reference architectures publish bandwidth per scalable unit rather than per array: the B300 'Enhanced' tier lists 250 GB/s read / 124 GB/s write for NVIDIA's 32-node, 256-GPU SU, scaling to 2,000 / 992 GB/s at eight SUs, against 80 / 40 and 640 / 320 for the Standard tier. For the Enhanced definition, 250 / 256 is ~0.98 GB/s/GPU. The separate 576-GPU SU is the GB200 NVL72 rack-scale planning unit; 250 / 576 is ~0.434 GB/s/GPU, but that juxtaposes different SU definitions and is not a valid B300 normalization. You size the tier to the matching SU count, not to a capacity quote.

Capacity sizing is the other axis, governed by a small set of ratios rather than a single number. The corpus itself (raw plus tokenized plus versioned), checkpoint retention (committed generations and in-flight space times the unique serialized state from Chapter 9.4), and a working-set multiplier for shuffled access together set the petabytes. A code/text run and a frontier multimodal run can need very different corpus footprints at the same GPU count. Corpus versions, replicas and checkpoint retention drive capacity; record them before buying single- or double-digit petabytes simply because a fleet has a thousand GPUs. Size the hot tier to bandwidth and the cold tier to corpus, and don't let a capacity datasheet push you into buying the hot tier by the petabyte.

Follow two concurrent jobs through cold reads, overlapping saves, restore and host loss. Size the complete bid from the governing constraint: this case buys extra capacity because endurance sets the drive count.

Select ten hosts and 120 drives on endurance. Require 100 GB/s reads and 40 GB/s protected logical writes simultaneously with one host absent and repair active, rounding the service floors upward. Nine surviving endpoints offer 9 × 40 = 360 GB/s per direction; qualification must establish how controllers, coding and repair consume that ceiling.

The BOM has 128 compute nodes: 256 compute + 20 storage + four metadata ports = 280 endpoint links. Compute placement 43/43/42, storage 5/5/0 and metadata 1/1/0 use 49/49/42 endpoint ports plus six uplinks on each plane's leaves. All fit 64 ports. The two planes need six leaves, six spines and 36 leaf-to-spine links; each spine uses six ports. Total cables: 280 + 36 = 316; cable-end terminations: 632. All-pluggable links need 632 modules; each DAC/AOC replaces its two module positions. Chapter 8.5 owns media, lengths, spares and port modes.

Each leaf's six 200 Gb/s uplinks supply 150 GB/s line rate per direction. Five storage hosts expose 100 GB/s per plane, leaving 50 GB/s before protocol and cross-leaf repair. Placing all 32 vision nodes and ten text nodes on leaf three demands (256 × 500 × 0.40)/1,000 + (80 × 20,000 × 4)/10⁹ ≈ 51 GB/s. Test encoded traffic against these cuts. This case covers one host loss; rack/site and plane-loss service require separate qualification.

Flip: surviving endurance permits 995.328/(1.5 × 1.1) ≈ 600 TB/day application writes, versus the assumed 509.6. Above that crossover, expand or change the qualified rating. The eight-host bid needs 840.84/(84 × 30.72) ≈ 0.33 DWPD, above 0.30. Extra namespace servers cannot repair a media-wear deficit. Complete the bid with metadata failover, firmware/power-loss behavior, GDS/client versions, licenses/support, OOB ports, racks, power, cooling and installation spares.

The storage subsystem does not live on its own wire. It shares — or must be deliberately kept from sharing — the fabric that carries the training collectives, and in a training cluster the most damaging storage mistakes are placement errors, not sizing errors. A checkpoint is a synchronized burst: thousands of ranks drain (write) or read at the same instant (Chapter 9.4). That is a textbook incast — many senders, converging traffic, transient congestion at the receiver's switch ports. Put that incast on the same fabric as your all-reduce, on the same cadence as your checkpoint interval, and you get a periodic collision that shows up as goodput loss correlated with checkpoint timing. The checkpoint succeeds every time; it just steals bandwidth from the collectives whenever it runs.

The placement decision has three options, and it is a co-design problem spanning storage and network rather than either alone (the fabric side is engineered in Chapter 8.5). A dedicated storage rail — a physically separate set of NICs and switches for storage traffic — means a checkpoint storm can never touch the back-end. Converging storage onto the back-end fabric with strict QoS is cheaper in NICs and switches, but you now rely on PFC/ECN priority classes and queue separation to keep the incast from starving the all-gather, and you must prove it holds under load. Sharing with no isolation works until the first concurrent checkpoint-and-collective collision, at which point step time degrades mysteriously and intermittently.

Data has gravity: the larger and more active a dataset, the harder and more expensive it is to move, and the more it pulls services and compute toward it. For an AI program this becomes a hard economic constraint at the petabyte scale, because the cloud bill for moving data — egress — is asymmetric by design. Ingress is free; egress is metered. The hyperscalers price internet egress at roughly $0.087–0.12/GB at the first paid tier (Azure ~$0.087, AWS ~$0.09, GCP Premium ~$0.12; list price, first tier — negotiated rates and the deeper volume tiers are lower), with cross-region and cross-AZ transfer adding their own per-GB charges on top. At petabyte scale these are not rounding errors: moving 1 PB out at the ~$0.09/GB first-tier rate is on the order of ~$90,000, and even at AWS's published volume tiers it is ~$54,000; a standard AI workload with weekly retraining can see egress reach 70–80% of the total cloud storage bill. Build the actual bill from stored GB-month by class, requests, retrieval bytes, each billed transfer leg, destination storage, verification and repeat reads after cache misses; AWS S3 pricing separates these components, and each candidate venue needs the same schedule on the same date.

That asymmetry forces the central question: move the data to the compute, or move the compute to the data? Move-data-to-compute is the default and is fine while the data is small or born where the GPUs are. But once the corpus is large, born in one place (a customer's region, a regulated jurisdiction, an on-prem lake), and accessed repeatedly, gravity flips the economics: bringing the accelerators to the data beats repeatedly paying egress to stream the data to a distant cluster. This is why frontier operators co-locate the prep supercomputer (Chapter 9.9) with the corpus, why regulated workloads pin compute to the jurisdiction where the data must legally remain (Chapter 10.10), and why multi-site strategy is downstream of gravity rather than the other way around.

The egress structure also creates a deliberate lock-in: because leaving costs money the incumbent keeps, gravity is a moat. The 2026 wrinkle is regulatory, and it has two separate halves that get conflated. Provider exit programs came first and are voluntary and conditional: AWS has waived egress for customers leaving the platform since March 2024, and the other majors run comparable programs you have to qualify for and apply to. The EU Data Act's outright ban on switching charges, egress included, bites only from 12 January 2027; until then providers may still levy cost-based switching fees. Either half dents the moat for a qualifying exit and does nothing for the day-to-day cross-region streaming that dominates an active AI pipeline. So design the data's birthplace and residence deliberately: land the corpus where the compute will live, replicate selectively rather than stream repeatedly, and treat every cross-region copy as a recurring egress liability, not a one-time cost.

Compare streaming and a local replica over one horizon. Streaming pays for charged cache-miss bytes on every pass plus both endpoint services. A replica pays for the initial verified move, changed-byte synchronization, protected capacity and operations. Initial move cost divided by positive recurring net savings gives the crossover horizon. Transfer bytes/sustained path payload plus verification and cutover gives the move deadline. Choose replication only if both tests pass; low reuse or high standing cost favors streaming. Data rights and recovery can exclude either option first.

Resilience in an AI storage subsystem is a different problem from resilience in enterprise IT, because the workload values a different thing. Traditional storage optimizes durability and availability — never lose a byte, never be unreachable. An AI training storage tier optimizes goodput: keep the GPUs fed and let a failure cost as few GPU-hours as possible (Chapter 12.2). The two diverge sharply on the hot tier. A scratch/checkpoint hot tier does not need eleven-nines durability — its contents are reproducible from the last durable checkpoint and the corpus — so heavy erasure coding and cross-site replication there buy durability the workload does not value, at the cost of the write bandwidth it does. The capacity tier is the inverse: the corpus and the canonical checkpoint history are the irreplaceable assets, and that is where durability spend belongs (Chapter 9.6).

Multi-tenancy and QoS come next, and this is where a shared storage tier earns or loses its keep. A storage fabric serving many tenants or many jobs is a noisy-neighbor problem in slow motion: one tenant's metadata storm or checkpoint burst can starve another's ingestion reads, and the victim sees it as unexplained GPU idle. The controls are tenant-level bandwidth and IOPS quotas, metadata-rate fairness, and priority classes that protect the latency-sensitive (KV, ingestion) from the bursty (checkpoint). This is the storage-tier mirror of the compute-side sharing spectrum in Chapter 10.3 and the congestion engineering in Chapter 8.6. Security rounds it out: encryption at rest and in flight, tenant isolation strong enough that one tenant cannot read another's corpus or checkpoints, and — for confidential workloads — attestation-gated access consistent with the TEE model in Chapter 11.5. Weights and training data are among the most valuable and most regulated assets in the building; the storage tier is where they sit at rest.

Storage is a small line on the bill of materials and a large lever on the outcome — which is exactly why it gets under-funded. The capex is real but bounded: an all-flash parallel filesystem sized to a large cluster’s per-GPU feed must be compared with the accelerator capex it keeps productive, including protected media, host service, fabric, software/support and installed power/cooling on the same duty boundary. The cost that dominates is the GPUs the storage strands. A hot tier one notch too small to keep a fleet fed, or a checkpoint path that perturbs collectives a few percent, costs goodput across the entire accelerator fleet — and on a power-bound, depreciating fleet, every point of goodput is a proportional fraction of the most expensive asset in the building. What you are pricing is the array plus the GPU-hours a wrong array burns.

The recurring costs split by tier and by venue. On-prem, the hot tier is dominated by flash media and the parallel-FS software/support; the capacity tier by $/TB and the power-and-space of dense flash or remaining HDD. In the cloud, egress and cross-region transfer can govern a repeatedly remote-read workload, as the gravity section showed, which is why a cloud storage TCO that ignores data movement is fiction. Price three things together: the media, the bandwidth (and the fabric it rides), and the movement. Optimize any one in isolation and the other two punish you.

Run the two-job workload and retain manifests, versions, topology and time series. Require cold loader rates, simultaneous 60 s drains and 5 s commits; repeat with one host absent and repair active at the 100/40 GB/s gates. In each matched repetition, loaded p99 collective step time must be ≤1.05× baseline. Above the assumed 5% crossover, choose more isolation, admission control or capacity.

Restore both jobs within Chapter 9.4's state-age, 180 s recovery and correct-next-step limits. Separately fail a namespace service and storage path against Chapter 9.2/Chapter 9.3 budgets. Test missing/corrupt generations, noisy neighbors and unauthorized reads; verify daily host writes and reserve. Procurement remains HOLD for these results and the installed power/cooling schedule. Chapter 13.9 owns commissioning execution.

Four trends are reshaping the sizing-and-resilience calculus, and each is treated in the consolidated roadmap (Chapter 16.2) rather than here. GPU- and DPU-initiated I/O moves the data path off the host CPU entirely: the accelerator (or the BlueField-class DPU) issues storage I/O directly, collapsing a CPU bottleneck that has capped the per-GPU feed and shifting where the storage rail terminates (Chapter 9.3). All-flash everywhere — QLC-backed object and capacity tiers displacing HDD — changes the capacity-tier TCO from a $/TB-and-spindle calculation to a $/TB-and-watt one, and narrows the bandwidth gap between hot and cold tiers (though the 2026 NAND shortage has paused the displacement on price — see Chapter 9.6). File/object convergence erodes the hard line between the parallel-FS hot tier and the object capacity tier, letting a single namespace span both and simplifying placement. And deeper CXL tiering (Chapter 9.7) inserts a memory-class tier between DRAM and NVMe that reshapes the inference KV hierarchy and, increasingly, the training scratch tier. None of these repeals the chapter's logic — you still size to a feed rate, isolate the incast, and pay for gravity. They move the numbers, not the decisions.

Choose the bid whose surviving configuration passes service, capacity, metadata and wear together, then site it using the complete movement bill and recovery boundary. The worked BOM buys more drives because endurance governs; another workload can instead buy uplinks, namespace services or a different cache policy. Accepting a paper total without the mixed-load and failure tests spends the same money while leaving the program's useful output unproven.