Parallel & Distributed File Systems

A parallel file system exists to keep accelerators saturated. The moment storage cannot feed the GPUs, the most expensive depreciating asset in the building sits idle while you pay for it (Chapter 9.1). This chapter covers the engine that feeds them: the parallel and distributed file system that fronts the NVMe flash, fans a single read out across hundreds of storage targets, and presents thousands of GPU clients with one coherent namespace. A wrong choice here shows up not as a missing feature but as paid GPU time lost to a starving data loader. Or it shows up as a checkpoint write that stalls a 16k-GPU synchronous job because the write tier cannot absorb the incast.

Five forks organize the chapter: POSIX vs object-native, centralized vs distributed metadata, erasure coding vs replication, open-source build vs certified appliance, and multiprotocol convergence vs tiered file-over-object. Each turns on a fact operators tend to learn the hard way. Once a small-file pipeline has enough sequential bandwidth, the limit is the metadata plane, and the workload that exposes it is lots of small files (LOSF), which dominates preprocessing, multimodal training and dataset curation. A platform striding through 5 billion 50 KB files at high IOPS is a different beast from one striding through a 10 GB sequential shard, yet the same datasheet number describes both.

The defining trick of a parallel file system is striping. A large file is split into stripes spread across many storage targets, so one client read pulls bandwidth from dozens of NVMe drives at once. Aggregate throughput then scales with the number of targets rather than the speed of any single drive. The data path and the metadata path scale differently, and that difference governs the whole system. Adding flash, targets and network grows sequential bandwidth roughly linearly. The metadata path does not scale that way, because every open(), stat(), readdir() and create() hits a service that must keep a consistent view of the namespace across thousands of concurrent clients. Because the two paths need different things from the network, decide in Chapter 9.3 whether storage gets dedicated links or shares the compute fabric. A shared namespace does not require shared switches, and a separate namespace does not reserve network capacity.

LOSF is therefore the benchmark that matters. A vision or multimodal dataset of billions of small files turns training into a metadata storm, where the data loader asks for millions of metadata operations per second rather than for bandwidth. A platform sized to the B300 reference's Enhanced tier (250 GB/s of sequential read per 32-node, 256-GPU NVIDIA scalable unit, against 80 GB/s for Standard) can still collapse to single-digit GPU utilization. That happens if its metadata server tops out at a few hundred thousand operations per second while the loader needs millions. The IO500 production list (as of ISC26, June 2026) shows how widely metadata results spread, but it is not a normalized head-to-head test. WEKA's Samsung SSC-24 entry (ISC25, 291 client nodes) scored about 2.75 million on IO500's metadata measure. The best Lustre entry, Helma (ISC25, 186 nodes), scored about 1.6 million, and the best DDN EXAScaler entry, HiPerGator (SC25, 10 nodes), about 0.94 million. The DAOS system at Argonne, Aurora (submitted at SC23, 300 nodes), scored about 103 million. Hardware, client counts, tuning and submission year all differ, and the metadata score is a geometric mean of mdtest and find phases, not a single operation rate. Use IO500 to shortlist, then normalize per client, per node and per dollar on your own LOSF trace before buying.

The five contenders differ mainly in what they make hard. Lustre/EXAScaler is the HPC incumbent. A Hyperion Research site survey, reported by HPCwire in October 2025, put it at 41% of HPC sites running a parallel file system, against 17% for Storage Scale and 6% for WEKA; no comparable share for AI clouds is published. It is also the most common file system on the IO500 lists, where DDN EXAScaler and HPE Lustre systems post many of the largest bandwidth results, behind the DAOS systems at the top. Legacy single-MDT deployments can bottleneck on metadata, while modern Lustre supports multiple metadata targets through Distributed Namespace (DNE), with HA/failover. An MDT is a logical target, not necessarily one physical server, so benchmark your deployment's MDT count, directory striping, placement, failover and small-file workload instead of labeling the product. GPFS/Storage Scale distributes metadata natively and brings the broadest multiprotocol and enterprise feature set, which is why it shows up in hyperscaler-validated GDS integrations (OCI + IBM Storage Scale).

WekaFS, now packaged as the NeuralMesh microservices architecture, was built software-first for the metadata-heavy, low-latency mixed I/O that AI generates. It spreads data and metadata across every node and is the platform most often cited for the LOSF win. VAST's DASE attacks a different axis: it separates stateless compute from flash so the two scale independently, and the data-lake and training tiers can collapse into one all-flash namespace (deep dive below). Ceph, DAOS and JuiceFS are the object-native answers: Ceph for unified on-prem block, object and file; DAOS for benchmark-leading raw HPC numbers; JuiceFS for file semantics over any S3 bucket in a cloud-native pipeline. A sixth route is standards-based parallel NFS (pNFS v4.2 with flexible-file layouts, for example Hammerspace), which uses the upstream Linux NFS client instead of a proprietary one. pNFS's first IO500 10-node production entry (SC25) reached about 75 GiB/s but only about 97 thousand on the metadata measure. Ten-node DDN EXAScaler entries score roughly 0.2 to 0.94 million, so test pNFS hardest on LOSF.

This fork decides whether a platform holds up at scale or wins a bake-off and then falls over in production. Centralized metadata (the classic Lustre MDS model) routes every namespace operation through one logical service. It is simpler to reason about, gives clean POSIX consistency, and suits large files read sequentially, the HPC pattern Lustre was born for. Distributed metadata (Weka, VAST, GPFS, and Lustre's DNE extension) shards the namespace across many nodes, so metadata throughput scales with the cluster instead of bottlenecking on one server.

The cost of getting it wrong is specific. Preprocessing and dataset curation are metadata-bound by nature: they walk directories, stat millions of files, dedup, shard and rewrite. Multimodal and vision training read billions of small image, audio and clip files. When the metadata plane saturates, the data loader cannot fill its prefetch queue, the GPUs drain their input buffers, and MFU falls off a cliff. Measure how far it falls by running the same workload with the loader fully supplied and comparing. Packing small files into large sequential shards (WebDataset, TFRecord, MDS, covered in Chapter 9.5) helps, but it also changes the namespace demand itself. If the packed format still gives the application the access it needs, a smaller metadata service can be enough. On Lustre, when files cannot be packed, two per-directory layout settings decide the LOSF result. Data-on-MDT keeps each small file's first extent on the MDT beside its inode, which removes the separate round trip to an OST, and DNE2 striped directories spread one hot directory across several MDTs. Ask which of these the bid benchmark used, and size MDT flash for the file data Data-on-MDT moves onto it. If LOSF still dominates, distributed placement and directory balance have to prove their worth in the test, so benchmark on your file-size distribution, not the vendor's hero sequential number.

Metadata demand starts at the loader. For each operation class j, count cold-path operations per sample o_j, multiply by admitted loaders L and samples/s per loader r, then apply the measured namespace-cache miss fraction m_j: λ_j = L × r × o_j × m_j. Add dataset creation, checkpoint publication, listing and cleanup as separate concurrent rates. Data IOPS do not convert into namespace capacity, and unlike benchmark scores do not add up. Count file opens per shard after packing and per sample before packing, because the format decision changes the demand.

Make the bid spell out directory placement, MDT or service shards, client and metadata caches, operation latency percentiles, namespace size, maximum admitted concurrency and remaining headroom. Replay the same operation mix through a metadata-service failover and against the hottest directory while payload reads, drains and rebuilds run. Compare completed operations/s and loader wait with the demand you calculated; an average across cold and cached runs hides startup failure. If the rate during failure misses demand, add services or change placement. If packing removes the demand without breaking random access or restart, pack instead. Record client and kernel versions, failover ownership and the exact file/object visibility tests alongside the result. The IO500 submission rules set the method boundaries; acceptance testing is in Chapter 9.8.

The 2026 trend that resolves much of this tension is multiprotocol single-namespace convergence. Data prep used to live on object (S3/HDFS), training on a POSIX parallel FS and inference somewhere else again, so data was copied between namespaces at every stage, at a cost in time, capacity and operational complexity. Weka and VAST now present file (POSIX/NFS/SMB), object (S3) and Kubernetes (CSI) views over a single namespace, and GPFS has long offered native S3 alongside POSIX. High-throughput object-over-flash (S3 Express-class) is fast enough to serve some training reads directly. The data path can stop being a copy farm: ingest as object, prep in place, train over POSIX against the same bytes, and serve inference from the same namespace.

This convergence rode into 2026 on a second trend, all-flash everywhere. QLC NVMe plus data reduction (dedup, compression, and VAST's similarity reduction) had narrowed the cost gap that once forced the capacity tier onto HDD. The NAND shortage then blew the gap back open. By Aug 2026, enterprise-SSD spot prices ran ~6.5× year-ago levels, putting a 30 TB QLC SSD at roughly 15× the price of a 30 TB nearline HDD (VDURA Flash Volatility Index; peak ~23× in Q1). NAND-maker capex diverted to DRAM/HBM limits new supply. All-flash everywhere is now a budget-and-lead-time decision rather than the default: the density and latency case for QLC stands, but the $/TB case does not until supply eases (TrendForce sketches possible 2H27 relief). So the choice is real. Pay for one multiprotocol all-flash platform that converges the tiers, or run a cheaper tiered file-over-object design and accept the copy tax and the operational seam between systems. → object economics and lifecycle tiering in Chapter 9.6; the data-loader path in Chapter 9.5.

How the durable tier protects data is a three-way tradeoff between capacity efficiency, rebuild cost and CPU/RAM overhead. Replication keeps N full copies (Ceph defaults to 3x). It is simple, fast to recover and trivial on CPU, but it triples your raw capacity for a given usable petabyte. Erasure coding splits each object into k data and m parity chunks; EC 4+2, for example, stores 1.5x the data for two-failure tolerance instead of 3x. That is dramatically more capacity-efficient, but every write and every recovery spends more RAM and CPU. A rebuild must read across many drives and recompute, so wide EC stripes lengthen rebuild windows and add load to a system that is already degraded. Drive size now sets the exposure window: a 122 TB QLC drive written at its 3.3 GB/s ceiling takes about 10 hours to rebuild onto one spare, and much longer when rebuild is throttled to protect training reads. Declustered or distributed-spare designs spread that write across many drives, so ask each vendor for measured time to full redundancy after one drive failure, at a stated foreground load.

On a hot training tier, where you want minimum rebuild impact and maximum throughput under failure, replication or a narrow declustered-RAID scheme is often the right call. On a capacity or object tier holding petabytes of training data and retained checkpoints, erasure coding can save a lot of raw capacity over replication, but the code you choose must still keep foreground traffic flowing during a rebuild. The platforms that win here (VAST's locally-decodable codes, Ceph EC pools) are engineered specifically to cap the rebuild penalty. Wide erasure coding on a latency-sensitive hot tier starves your GPUs during a rebuild. Replication at 3x on a 50 PB cold lake doubles your capacity bill for resilience the workload does not value.

The last fork is commercial, and it is where fast growth in AI storage and ongoing vendor consolidation bite. Open-source build means running Lustre, Ceph or DAOS on your own hardware. It gives you the lowest licensing cost, full control and no per-terabyte vendor tax. The price is owning the operations: tuning the metadata servers, managing rebuilds, chasing kernel and client compatibility, and being your own support desk when a 16k-GPU job hangs on a storage stall at 3 a.m. DAOS holds the top IO500 production results (Argonne's Aurora and LRZ's SuperMUC-NG Phase 2 on the ISC26 list). Since Intel moved its DAOS engineering team to HPE in December 2024, DAOS has commercial routes: HPE's Cray K3000 storage system, and Enakta Labs' supported platform with SMB, NFS and S3 gateways. Its installed base, tooling and multiprotocol features are still far thinner than Storage Scale, Weka or VAST, so an AI cloud that adopts it should budget for that gap. Certified appliances (DDN, Weka, VAST, IBM) cost more and carry lock-in, but they ship certified against the GPU vendor's reference architecture. That buys validated per-scalable-unit bandwidth tiers, supported GPUDirect Storage paths, and a vendor on the hook when something breaks.

Certification is what you are paying for. NVIDIA's DGX SuperPOD reference architectures specify exact storage bandwidth tiers per scalable unit. The B300 'Enhanced' tier is 250 GB/s read / 124 GB/s write for a 32-node, 256-GPU NVIDIA SU, or about 0.98 GB/s/GPU read. A certified platform has been measured hitting those targets over a supported GDS data path. If you build your own, proving you meet those targets is your job, and the cost of being wrong is a fleet that under-feeds. The decision rule: at frontier scale with a long-lived, well-staffed storage team, open-source can be the right economic call. For most operators, buying a certified appliance and treating the premium as insurance against GPU-idle risk is rational. Either way, settle before signing which reference architecture and which GDS path the certification covers; a vague "AI-ready" claim is not certification. → GPUDirect Storage and the data path in Chapter 9.3.

The common mis-scopes in parallel-FS selection all come from optimizing the visible number instead of the one that limits the workload:

• Buying on sequential bandwidth for a metadata-bound workload. A stunning GB/s number and a mediocre metadata score, bought for an LOSF-dominated preprocessing or multimodal pipeline. The GPUs then wait on the data loader for part of every step, on data you pay full price to host. Replay your own file-size mix to see how much, and benchmark metadata on it before bandwidth.
• One namespace for every personality. Forcing the metadata storm of data prep, the sequential throughput of training and the latency of KV-cache onto a single tier because convergence sounded cheaper. The four I/O personalities pull in opposite directions. Converge where it pays (file + object on the training/lake tier), but do not pretend one tier serves all four well. → Chapter 9.1.
• Wide erasure coding on the hot tier. Capacity-efficient until a drive fails mid-run, when the rebuild read-amplification starves the very GPUs the tier exists to feed. Keep wide EC for a cold or object tier whose repair traffic still leaves enough foreground performance. Use replication or narrow declustered RAID where their measured rebuild impact fits your degraded-state budget better.
• Uncertified "AI-ready" storage behind a non-blocking fabric. Pairing a multi-million-dollar GPU cluster with a platform never measured against your GPU vendor's SU bandwidth tiers, then discovering at scale that it cannot sustain the GDS path. Certification against a specific reference architecture is the cheap insurance.

Choose the interface and metadata placement that survive your namespace trace, then buy the support model your team can run through a failed controller or metadata service. On a multiprotocol platform, write a file through one advertised protocol, then read, overwrite and delete it through the others with the permissions they should have. CSI provisioning alone does not prove the protocols see the same bytes. A certified appliance cuts integration work but still has to pass the workload test. A self-managed stack saves the appliance premium only if you also fund the staffing, maintenance and recovery it needs.