Parallel & Distributed File Systems

Chapter 9.1 established the economic argument: a parallel file system exists to keep accelerators saturated, and the moment storage cannot feed the GPUs, you are burning the most expensive depreciating asset in the building at idle. This chapter is about the engine that does the feeding — the parallel and distributed file system that fronts the NVMe flash, fans a single read out across hundreds of storage targets, and presents thousands of GPU clients with one coherent namespace. It is the layer where a wrong decision does not show up as a missing feature; it shows up as a fleet running at 50% MFU because the data loader is starving, or as a checkpoint write that stalls a 16k-GPU synchronous job because the write tier cannot absorb the incast.

The forks in this chapter are POSIX vs object-native, centralized vs distributed metadata, erasure coding vs replication, open-source build vs certified appliance, and multiprotocol convergence vs tiered file-over-object. The unifying insight, and the one most operators learn the hard way, is that AI storage is won or lost on the metadata plane, not on sequential bandwidth: every modern all-flash platform can saturate a few hundred GB/s per scalable unit. The binding constraint is the lots-of-small-files (LOSF) workload that dominates preprocessing, multimodal training, and dataset curation. The platform that strides through 5 billion 50 KB files at high IOPS is a different beast from the one that strides through a 10 GB sequential shard, and the same datasheet number describes both.

The defining trick of a parallel file system is striping: a single file is split into stripes spread across many storage targets, so one client read pulls bandwidth from dozens of NVMe drives at once and aggregate throughput scales with the number of targets, not the speed of any single drive. The data path and the metadata path are architecturally separate, and that separation governs how the system scales. The data path is easy to scale: add flash, add targets, add network, and sequential bandwidth grows roughly linearly. The metadata path is hard, because every open(), stat(), readdir(), and create() hits a service that must maintain a consistent view of the namespace across thousands of concurrent clients.

This is why the LOSF workload is the real benchmark. A vision or multimodal dataset of billions of small files turns training into a metadata storm: the data loader is not asking for bandwidth, it is asking for millions of metadata operations per second. A platform sized to 250 GB/s of sequential read per scalable unit can still collapse to single-digit GPU utilization if its metadata server tops out at a few hundred thousand operations per second while the loader needs millions. The published gap is stark: in 2025 IO500 metadata measurements, Weka demonstrated on the order of 2.75 million metadata IOPS against roughly 520,000 for a comparable DDN/Lustre configuration — a ~5x advantage that is invisible on a bandwidth datasheet but decisive for an AI training pipeline (Blocks & Files / StorageMath, IO500 SC25). When you evaluate a platform, you are evaluating its metadata plane first and its bandwidth second.

The contenders differ in consequence, not rank. Lustre/EXAScaler is the HPC incumbent — roughly 41% of parallel-FS deployments in HPC surveys — and it is unmatched at raw sequential throughput, which is why DDN systems dominate the IO500 bandwidth lists. Its historical weakness is the single metadata server; modern Lustre mitigates this with Distributed Namespace (DNE) and multiple metadata targets, but you are explicitly engineering around a centralized design rather than getting distribution for free. GPFS/Storage Scale (~17% share) distributes metadata natively and brings the broadest multiprotocol and enterprise-feature set, which is why it shows up in hyperscaler-validated GDS integrations (OCI + IBM Storage Scale). WekaFS, now packaged as the NeuralMesh microservices architecture, was built software-first for exactly the metadata-heavy, low-latency mixed I/O that AI generates — it distributes both data and metadata across every node and is the platform most often cited for the LOSF win. VAST's DASE attacks a different axis: it disaggregates stateless compute (CNodes) from flash enclosures (DNodes) over NVMe-oF and puts a single global ACID element store underneath, which lets it scale capacity and compute independently and collapse the data-lake and training tiers into one all-flash namespace. Ceph, DAOS, and JuiceFS are the object-native answers — Ceph for unified on-prem block/object/file, DAOS for benchmark-leading raw HPC numbers, JuiceFS for file semantics over any S3 bucket in a cloud-native pipeline.

This is the fork that separates a platform that works at scale from one that wins a bake-off and then falls over in production. Centralized metadata (the classic Lustre MDS model) routes every namespace operation through one logical service. It is simpler to reason about, gives clean POSIX consistency, and is fine for a workload of large files read sequentially — the HPC pattern Lustre was born for. Distributed metadata (Weka, VAST, GPFS, and Lustre's DNE extension) shards the namespace across many nodes so metadata throughput scales with the cluster instead of bottlenecking on one server.

The downstream cost of getting this wrong is specific and measurable. Preprocessing and dataset curation are metadata-bound by nature — they walk directories, stat millions of files, dedup, shard, and rewrite. Multimodal and vision training read billions of small image/audio/clip files. When the metadata plane saturates, the data loader cannot fill its prefetch queue, the GPUs drain their input buffers, and MFU falls off a cliff — the naive small-file path commonly runs under 50% GPU utilization versus over 90% with a well-fed loader. The fix is partly a data-format decision (pack small files into large sequential shards — WebDataset, TFRecord, MDS — covered in Chapter 9.5), but the format only papers over a metadata plane that cannot keep up. If your dominant workload is LOSF, distributed metadata is not a nice-to-have; it is the entry ticket, and you should benchmark it on your file-size distribution, not the vendor's hero sequential number.

The 2026 architectural trend that resolves much of this tension is multiprotocol single-namespace convergence. Historically the data-prep pipeline lived on object (S3/HDFS), the training tier lived on a POSIX parallel FS, and inference pulled from somewhere else again — which meant data was copied between namespaces at every stage, paying a tax in time, capacity, and operational complexity. Weka and VAST now present file (POSIX/NFS/SMB), object (S3), and Kubernetes (CSI) views over a single namespace; GPFS has long offered native S3 alongside POSIX; and high-throughput object-over-flash (S3 Express-class) is fast enough to serve some training reads directly. The consequence is that the data path can stop being a copy farm: ingest as object, prep in place, train over POSIX against the same bytes, and serve inference from the same namespace.

This convergence is enabled by the second 2026 trend — all-flash everywhere. QLC NVMe plus data reduction (dedup, compression, and VAST's similarity reduction) has collapsed the cost gap that historically forced the capacity tier onto HDD. When the capacity tier is also flash, the economic argument for keeping the lake and the training tier on separate media weakens, and a single converged platform becomes viable. The fork is now genuine: pay for one multiprotocol all-flash platform that converges the tiers, or run a cheaper tiered file-over-object design and accept the copy tax and the operational seam between systems. → object economics and lifecycle tiering in Chapter 9.6; the data-loader path in Chapter 9.5.

How the durable tier protects data is a three-way tradeoff between capacity efficiency, rebuild cost, and CPU/RAM overhead. Replication keeps N full copies (Ceph defaults to 3x) — simple, fast to recover, trivial on CPU, but it triples your raw capacity for a given usable petabyte. Erasure coding splits each object into k data and m parity chunks (e.g. EC 4+2 stores 1.5x the data for two-failure tolerance instead of 3x), which is dramatically more capacity-efficient but spends more RAM and CPU on every write and, critically, on every recovery — a rebuild must read across many drives and recompute, so wide EC stripes lengthen rebuild windows and raise the load on a degraded system.

For AI, the decision interacts with the all-flash and failure-domain story. On a hot training tier where you want minimum rebuild impact and maximum throughput under failure, replication or a narrow declustered-RAID scheme is often the right call. On a capacity or object tier holding petabytes of training data and retained checkpoints, the 2x capacity savings of erasure coding is too large to ignore, and the platforms that win here (VAST's locally-decodable codes, Ceph EC pools) are specifically engineered to cap the rebuild penalty. The wrong default — wide erasure coding on a latency-sensitive hot tier, or 3x replication on a 50 PB cold lake — either starves your GPUs during a rebuild or doubles your capacity bill for no resilience the workload values.

The last fork is commercial, and it is where the parallel-FS market's ~24% CAGR and ongoing consolidation bite. Open-source build — running Lustre, Ceph, or DAOS on your own hardware — gives you the lowest licensing cost, full control, and no per-terabyte vendor tax, at the price of owning the operational burden: tuning the metadata servers, managing rebuilds, chasing kernel and client compatibility, and being your own support desk when a 16k-GPU job hangs on a storage stall at 3 a.m. DAOS posts spectacular IO500 numbers (Argonne and LRZ systems led the SC25 production list) but is candidly a niche HPC system that lacks much of the enterprise feature set and support an AI cloud operator expects. Certified appliances (DDN, Weka, VAST, IBM) cost more and carry lock-in, but they ship certified against the GPU vendor's reference architecture — validated per-scalable-unit bandwidth tiers, supported GPUDirect Storage paths, and a vendor on the hook when something breaks.

Certification is what you are actually paying for. NVIDIA's DGX SuperPOD reference architectures specify exact storage bandwidth tiers per scalable unit (the B300 'Enhanced' tier of 250 GB/s read / 124 GB/s write per SU), and a certified platform is one that has been measured to hit them with a supported GDS data path. Building your own means you also own proving you meet those targets — and the cost of being wrong is a fleet that under-feeds. The honest decision rule: at frontier scale with a long-lived, well-staffed storage team, open-source can be the right economic call; for most operators buying a certified appliance and treating the premium as insurance against GPU-idle risk is rational. Either way, the question to answer before signing is which specific reference architecture and which GDS path you are certifying against — vague "AI-ready" claims are not certification. → GPUDirect Storage and the data path in Chapter 9.3.

The recurring mis-scopes in parallel-FS selection all come from optimizing the visible number instead of the binding constraint:

• Buying on sequential bandwidth for a metadata-bound workload. The classic error: a platform with a stunning GB/s number and a mediocre metadata score, bought for a preprocessing or multimodal pipeline that is LOSF-dominated. The fleet runs at half its MFU on data it pays full price to host. Benchmark metadata on your file-size distribution first.
• One namespace for every personality. Forcing the metadata storm of data-prep, the sequential throughput of training, and the latency of KV-cache onto a single tier because convergence sounded cheaper. The four I/O personalities pull in opposite directions; converge where it pays (file+object on the training/lake tier) but do not pretend one tier serves all four well. → Chapter 9.1.
• Wide erasure coding on the hot tier. Capacity-efficient until a drive fails mid-run, when the rebuild read-amplification starves the very GPUs the tier exists to feed. Keep wide EC for the cold/object tier; use replication or narrow declustered RAID where rebuild impact must stay bounded.
• Uncertified "AI-ready" storage behind a non-blocking fabric. Pairing a platform that has never been measured against your GPU vendor's SU bandwidth tiers with a multi-million-dollar GPU cluster, then discovering at scale that it cannot sustain the GDS path. Certification against a specific reference architecture is the cheap insurance.