Sizing, Data Gravity & Resilience

By the time you reach this chapter you have the pieces: the parallel filesystem (Chapter 9.2), the NVMe data path (Chapter 9.3), the checkpoint math (Chapter 9.4), the loader (Chapter 9.5), the object tier (Chapter 9.6), and the inference KV hierarchy (Chapter 9.7). This chapter assembles them into a sized, sited, survivable subsystem. The forks here are unusually unforgiving because storage failure is invisible in the worst way. A mis-sized fabric throws an error; a mis-sized storage tier does not fail — it simply starves the GPUs, and the loss appears as a utilization number a few points below where it should be, indistinguishable at a glance from a hundred other causes.

Four decisions structure the chapter. Sizing: the storage:compute ratio and the bandwidth budget that decide whether the accelerators stay fed. Network co-design: how the checkpoint incast is kept off the training collectives, which is the single most consequential placement decision in the storage subsystem. Data gravity: whether you move the corpus to the compute or the compute to the corpus, and the egress economics that govern multi-site strategy. Resilience and TCO: durability, multi-tenancy QoS, security, and the benchmarking-and-acceptance regime that turns a vendor claim into a contractual commitment. Each is a place where a strategist's choice and an engineer's number meet.

The first and most consequential fork is the one Chapter 9.1 opened: is each tier bandwidth-sized or capacity-sized? Almost every AI hot tier is bandwidth-sized — its job is to keep accelerators fed, and the governing number is GB/s per GPU, not petabytes. Almost every AI capacity tier is capacity-sized — its job is to hold the corpus and the checkpoint history, and the governing number is cost per TB. Size the hot tier by capacity and you buy a vast array that cannot saturate the GPUs; size the capacity tier by bandwidth and you overpay for flash speed the cold corpus never touches. The error in either direction strands the asset you under-prioritized.

The hot-tier budget starts from a per-GPU read target. NVIDIA's reference floor is ~1 GB/s/GPU; vision and multimodal training references run ~4 GB/s/GPU read; checkpoint-heavy and shuffle-heavy pipelines push 4–10 GB/s/GPU. Multiply by GPU count and you have an aggregate read requirement; apply NVIDIA's write ≥ ½ read rule and you have the write floor that the checkpoint drain must clear. This is why the SuperPOD reference architectures publish bandwidth per scalable unit rather than per array: the B300 'Enhanced' tier lists ~250 GB/s read / ~124 GB/s write per SU, scaling to ~2,000 / ~992 GB/s at eight SUs. You size the tier to the SU count, not to a capacity quote.

Capacity sizing is the other axis, and it is governed by a small set of ratios rather than a single number. The corpus itself (raw plus tokenized plus versioned), the checkpoint retention depth (how many historical checkpoints you keep, times ~14 bytes/param/checkpoint), and a working-set multiplier for shuffled access together set the petabytes. Practitioner ratios for storage-to-compute capacity cluster in the single-digit-to-low-double-digit PB per thousand GPUs for large pre-training, but the honest answer is that the corpus drives it and the corpus is workload-specific — a code/text run and a frontier multimodal run differ by an order of magnitude. The discipline is to size the hot tier to bandwidth and the cold tier to corpus, and to never let a capacity datasheet talk you into buying the hot tier by the petabyte.

The storage subsystem does not live on its own wire. It shares — or must be deliberately kept from sharing — the fabric that carries the training collectives, and the most damaging storage mistake in a training cluster is a placement error, not a sizing error. A checkpoint is a synchronized burst: thousands of ranks drain (write) or read at the same instant (Chapter 9.4). That is a textbook incast — many senders, converging traffic, transient congestion at the receiver's switch ports. Put that incast on the same fabric as your all-reduce, on the same cadence as your checkpoint interval, and you have engineered a periodic collision that shows up as goodput loss correlated with checkpoint timing. The checkpoint succeeds every time; it just steals bandwidth from the collectives whenever it runs.

The fork is a placement decision with three options, and it is a co-design problem spanning storage and network rather than either alone (the fabric side is engineered in Chapter 8.5). You can run a dedicated storage rail — a physically separate set of NICs and switches for storage traffic, so a checkpoint storm can never touch the back-end. You can converge storage onto the back-end fabric with strict QoS — cheaper in NICs and switches, but you are now relying on PFC/ECN priority classes and queue separation to keep the incast from starving the all-gather, and you must prove it holds under load. Or you can share with no isolation — which works until the first concurrent checkpoint-and-collective collision, at which point step time degrades mysteriously and intermittently.

Data has gravity: the larger and more active a dataset, the harder and more expensive it is to move, and the more it pulls services and compute toward it. For an AI program this stops being a metaphor at the petabyte scale and becomes a hard economic constraint, because the cloud bill for moving data — egress — is asymmetric by design. Ingest is free; egress is metered. The hyperscalers price internet egress at roughly $0.087–$0.12/GB at the first tier in 2026 (Azure ~$0.087, AWS ~$0.09, GCP Premium ~$0.12), with cross-region and cross-AZ transfer adding their own per-GB charges on top. At petabyte scale these are not rounding errors: moving 1 PB out at ~$0.09/GB is on the order of ~$90,000, and a standard AI workload with weekly retraining can see egress reach 70–80% of the total cloud storage bill (EgressCost.com, 2026).

That asymmetry forces the central gravity fork: move the data to the compute, or move the compute to the data? Move-data-to-compute is the default and is fine while the data is small or born where the GPUs are. But once the corpus is large, born in one place (a customer's region, a regulated jurisdiction, an on-prem lake), and accessed repeatedly, gravity flips the economics: it is cheaper and faster to bring the accelerators to the data than to repeatedly pay egress to stream the data to a distant cluster. This is why frontier operators co-locate the prep supercomputer (Chapter 9.9) with the corpus, why regulated workloads pin compute to the jurisdiction where the data must legally remain (Chapter 10.10), and why multi-site strategy is downstream of gravity rather than the other way around.

The egress structure also creates a deliberate lock-in: because leaving costs money the incumbent keeps, gravity is a moat. The 2026 wrinkle is regulatory — under the EU Data Act the major clouds now waive egress for customers who are leaving the platform, which dents the moat for exits but not for the day-to-day cross-region streaming that dominates an active AI pipeline. The strategic move is to design the data's birthplace and residence deliberately: land the corpus where the compute will live, replicate selectively rather than stream repeatedly, and treat every cross-region copy as a recurring egress liability, not a one-time cost.

Resilience in an AI storage subsystem is not the same problem as resilience in enterprise IT, because the workload values a different thing. Traditional storage optimizes durability and availability — never lose a byte, never be unreachable. An AI training storage tier optimizes goodput: keep the GPUs fed and let a failure cost as few GPU-hours as possible (Chapter 12.2). The two diverge sharply on the hot tier. A scratch/checkpoint hot tier does not need eleven-nines durability — its contents are reproducible from the last durable checkpoint and the corpus — so spending on heavy erasure coding and cross-site replication there buys durability the workload does not value, at the cost of the write bandwidth it does. The capacity tier is the inverse: the corpus and the canonical checkpoint history are the irreplaceable assets, and that is where durability spend belongs (Chapter 9.6).

Multi-tenancy and QoS is the next axis, and it is where the shared storage tier earns or loses its keep. A storage fabric serving many tenants or many jobs is a noisy-neighbor problem in slow motion: one tenant's metadata storm or checkpoint burst can starve another's ingestion reads, and the victim sees it as unexplained GPU idle. The controls are tenant-level bandwidth and IOPS quotas, metadata-rate fairness, and priority classes that protect the latency-sensitive (KV, ingestion) from the bursty (checkpoint). This is the storage-tier mirror of the compute-side sharing spectrum in Chapter 10.3 and the congestion engineering in Chapter 8.6. Security closes the set: encryption at rest and in flight, tenant isolation strong enough that one tenant cannot read another's corpus or checkpoints, and — for confidential workloads — attestation-gated access consistent with the TEE model in Chapter 11.5. Weights and training data are among the most valuable and most regulated assets in the building; the storage tier is where they sit at rest.

Storage is a small line on the bill of materials and a large lever on the outcome, which is exactly the asymmetry that gets it under-funded. The capex is real but bounded — an all-flash parallel filesystem sized to the per-GPU feed of a large cluster is a single-digit percentage of the GPU capex it serves. The consequential cost is not the storage; it is the GPUs the storage strands. A hot tier sized one notch too small to keep a fleet fed, or a checkpoint path that perturbs collectives a few percent, costs goodput on the entire accelerator fleet — and on a power-bound, depreciating fleet, every point of goodput is a proportional fraction of the most expensive asset in the building. The TCO calculation that matters is not the price of the array; it is the price of the array plus the GPU-hours a wrong array burns.

The recurring costs split by tier and by venue. On-prem, the hot tier is dominated by flash media and the parallel-FS software/support; the capacity tier by $/TB and the power-and-space of dense flash or remaining HDD. In the cloud, the headline storage rate is rarely the largest number — egress and cross-region transfer are, as the gravity section showed, which is why a cloud storage TCO that ignores data movement is fiction. The honest model prices three things together: the media, the bandwidth (and the fabric it rides), and the movement. Optimize any one in isolation and the other two punish you.

Four trends are reshaping the sizing-and-resilience calculus, and each is treated in the consolidated roadmap (Chapter 16.2) rather than here. GPU- and DPU-initiated I/O moves the data path off the host CPU entirely — the accelerator (or the BlueField-class DPU) issues storage I/O directly, collapsing a CPU bottleneck that has capped the per-GPU feed and shifting where the storage rail terminates (Chapter 9.3). All-flash everywhere — QLC-backed object and capacity tiers displacing HDD — changes the capacity-tier TCO from a $/TB-and-spindle calculation to a $/TB-and-watt one, and narrows the bandwidth gap between hot and cold tiers. File/object convergence erodes the hard line between the parallel-FS hot tier and the object capacity tier, letting a single namespace span both and simplifying placement. And deeper CXL tiering (Chapter 9.7) inserts a memory-class tier between DRAM and NVMe that reshapes the inference KV hierarchy and, increasingly, the training scratch tier. None of these repeals the chapter's logic: you still size to a feed rate, still isolate the incast, still pay for gravity. They change the numbers, not the forks.