Storage & Data

Storage decides whether an accelerator computes or idles, and getting it right means designing for four competing I/O personalities that can exhaust different resources on the same flash and fabric.

A parallel file system keeps a GPU saturated only if its metadata plane survives the workload’s small-file storm and its data plane supplies the bytes; test billions of small files when the corpus contains them, because sequential bandwidth alone cannot qualify that purchase.

Whether a GPU stays fed can turn on per-request latency before aggregate bandwidth; a supported direct path routes bytes from flash straight into HBM without a host-memory bounce buffer, but the measured workload and surviving path decide whether that bypass earns its cost.

Checkpointing is a training cluster's goodput control knob — the interval, tier, and write bandwidth you choose decide how many GPU-hours each failure erases, and at frontier scale failures arrive constantly.

The data loader sits in the critical loop of every training step; get its format, sharding, and CPU budget wrong and the GPUs stall, burning depreciation while they wait to be fed.

Object storage holds the whole corpus, every checkpoint lineage, and every shipped model; its cost turns less on vendor choice than on whether you build it as a flash-fronted serving layer the GPUs read from.

Inference made the KV-cache a first-class storage problem: its bytes now spill past HBM, and where you let them land — DRAM, CXL, NVMe, or Ethernet-flash — sets your tokens-per-second and cost-per-token.

Size the hot tier by per-GPU bandwidth when sequential delivery binds, check metadata, retained bytes and endurance, place the corpus where the compute can reach it, and control checkpoint incast — miss a budget in the surviving system and idle accelerators absorb the cost.

Before any GPU sees a token, a second cluster must dedupe, filter, decontaminate, and tokenize trillions of tokens — undersize it and you starve the training fleet or waste GPU-hours on string processing.