MLOps & Training Frameworks

Everything else in Part 10 gets electrons to the GPU, keeps the node healthy, and recovers it when it dies. This chapter is about what runs on that fleet to turn its arithmetic into a trained model — and it is where a power-bound, depreciation-clocked data center either earns its megawatts or wastes them. A cluster can be perfectly provisioned (Chapter 10.5), perfectly scheduled (Chapter 10.1), and perfectly observed (Chapter 10.6), and still run at 30% model-FLOPS-utilization because someone chose the wrong parallelism layout or let checkpoint overhead eat a fifth of every run. The framework and the control plane are the conversion efficiency between the capex you bought and the weights you ship.

Three choices dominate. Which training framework you standardize on (FSDP2 vs DeepSpeed/ZeRO vs Megatron-Core) determines which parallelism strategies you can express cheaply and how much of your team's time goes to plumbing instead of research. Which orchestration plane owns the job lifecycle (Kubernetes-native vs Ray vs SkyPilot-over-Slurm) determines how gang scheduling, topology placement, and elastic recovery behave when — not if — a node fails. And how you handle reproducibility and checkpointing determines whether a run is a scientific artifact you can re-derive or a one-off you can never explain. Inference serving — vLLM, TensorRT-LLM, SGLang, disaggregation, goodput-optimal scheduling — is a different discipline with different SLOs and is owned end-to-end in Chapter 10.11; this chapter stops at the trained checkpoint.

A single accelerator cannot hold a frontier model. A dense 70B model in BF16 needs ~140 GB just for weights, and training state — optimizer moments, gradients, master weights — pushes the real footprint to roughly 16 bytes per parameter with mixed-precision Adam (2 bytes BF16 weight + 2 bytes BF16 gradient + 12 bytes FP32 master weights and Adam moments) — two bytes per parameter more than the ~14 bytes/parameter checkpoint recipe in Chapter 9.4, which leaves the gradient out because a restart recomputes it. Even a GB200 GPU with 186 GB usable HBM, or the larger HBM3E stacks on GB300-class parts, cannot hold a 70B+ training job, let alone a 400B+ one. The framework's first job is to shard that state across the fleet; its second is to do so while keeping the GPUs busy. Three families dominate production in 2026, and they sit on a spectrum from 'least code, least control' to 'most code, most control.'

PyTorch FSDP2 (Fully Sharded Data Parallel, the rewritten v2 that replaced the FlatParameter model with per-parameter DTensor sharding) is the path of least friction for PyTorch-native shops. It shards parameters, gradients, and optimizer state across the data-parallel group and reconstitutes each layer's full weights on demand via an all-gather that prefetches the next shard while computing on the current one — the same idea as DeepSpeed ZeRO-3, expressed natively in the framework. FSDP2 is the right default for mid-scale training (single-digit-thousands of GPUs) and for teams that want minimal abstraction over their model code. Its weakness is that pure parameter-sharding is a data-parallel technique: once you need tensor or pipeline parallelism to break a single layer across GPUs, FSDP2 alone is not enough, and you compose it with PyTorch's device-mesh primitives or reach for Megatron.

DeepSpeed/ZeRO (Microsoft) pioneered the partition-the-redundancy idea: ZeRO-1 shards optimizer state, ZeRO-2 adds gradient sharding, ZeRO-3 adds parameter sharding (the FSDP-equivalent). Its durable niche in 2026 is the offload-heavy lane — ZeRO-Infinity streams parameters and optimizer state to CPU DRAM and NVMe, letting a memory-starved cluster train a model that would not otherwise fit. That is a goodput-for-fit trade: you accept lower MFU (the offload traffic competes with compute) in exchange for running at all on the hardware you have. DeepSpeed also retains strong MoE support and a large installed base, so 'we already have DeepSpeed expertise' is itself a legitimate reason to stay.

Megatron-Core (NVIDIA) is the frontier-scale framework: it is where you go when each gained absolute MFU point reduces the GPU-hours required for a fixed training workload across the entire fleet. Megatron implements true model parallelism — tensor parallelism (splitting individual matmuls across GPUs), pipeline parallelism (splitting layers into stages), and now expert and context parallelism — with hand-tuned communication-overlap kernels. It is the most code and the most operational burden, and it rewards that burden with the highest realized utilization at the largest scales. The 2026 reality is that these families increasingly compose rather than compete: Megatron-FSDP and TorchTitan let you layer FSDP-style sharding under Megatron's tensor/pipeline/expert dimensions, and most frontier labs run a hybrid rather than a single pure framework.

Once you have a framework, the decisive choice is the parallelism layout: how you map the model and the batch onto the physical fleet across what used to be three dimensions and is now four or five. Each dimension trades communication for memory along a different axis, and the optimal mapping is a function of your scale-up domain size, your back-end fabric, and the model's shape. There is no portable default; a layout tuned for 8-GPU HGX nodes is wrong on a 72-GPU NVL72 domain.

• Data parallelism (DP): replicate the model, split the batch, all-reduce gradients each step. Cheapest to reason about, but every replica needs the whole model — which is why it is paired with sharding (FSDP/ZeRO) to remove the redundancy.
• Tensor parallelism (TP): split an individual matmul (and its layer) across GPUs. Communication is heavy and latency-sensitive (all-reduce inside every layer), so TP must stay inside the scale-up domain — NVLink, not the scale-out fabric. This is the dimension that most directly couples the layout to hardware: a 72-GPU NVLink domain lets you run wider TP than an 8-GPU node, which is why bigger scale-up domains matter (Chapter 8.5).
• Pipeline parallelism (PP): split the layers into stages, each on a different GPU group, and stream micro-batches through. Cheap communication (only activations cross stage boundaries) but introduces the pipeline 'bubble' — idle time at fill and drain — which interleaved schedules (1F1B, zero-bubble variants) exist to minimize.
• Expert parallelism (EP): for Mixture-of-Experts models, place different experts on different GPUs and route tokens to them via all-to-all. The all-to-all is the bottleneck, and a wide NVLink domain (EP32 on NVL72 vs EP8 on a node) is what makes wide expert parallelism throughput-positive rather than a network disaster (Chapter 8.5).
• Context parallelism (CP): split along the sequence-length dimension so a single very-long-context example fits. The fifth dimension, and increasingly necessary as context windows grow.

Getting the layout wrong has concrete, compounding costs. TP that spills out of the NVLink domain onto the scale-out fabric collapses MFU because the latency-sensitive in-layer all-reduce now traverses a slower hop. PP with too few micro-batches spends a large fraction of each step in the bubble. EP that is too wide for the fabric saturates the all-to-all. The layout is where the abstract 'parallelism strategy' meets the concrete topology of the building, and it is the tuning decision that moves MFU most.

A framework runs one job. The orchestration plane runs the fleet of jobs: it places them topology-aware, gang-schedules every rank so a job either gets all its GPUs or none (a half-placed synchronous job is a deadlock, not a slow start), preempts and re-queues, and re-forms the job when ranks die. This is the boundary between this chapter and the scheduling plane of Chapter 10.1 and Chapter 10.2: the scheduler decides where a job lands; the training-orchestration layer above it decides how the job's lifecycle is driven and how the pipeline of pre-processing, training, eval, and checkpointing is wired. Three patterns dominate.

Kubernetes-native (Kubeflow Trainer on top of a gang-capable scheduler — Volcano, Kueue, or NVIDIA's KAI — with topology-aware placement and increasingly Dynamic Resource Allocation for GPUs) is the path for operators who already run Kubernetes and want training to be one more workload on the same control plane. Kubeflow Trainer's 2026 line added distributed data cache and topology-aware scheduling via Kueue/Volcano. The strength is operational unification — one cluster, one RBAC model, one observability stack. The weakness is that vanilla Kubernetes was never built for gang-scheduled, topology-sensitive, all-or-nothing jobs, which is exactly why Volcano/Kueue/KAI exist as a layer to retrofit that behavior.

Ray is the application-framework answer: an actor/task runtime that expresses distributed training, RL rollouts, and data pipelines as Python and handles fine-grained orchestration within a job. The 2026 norm is hybrid — Kubernetes (or KubeRay) for coarse-grained resource management and Ray for the fine-grained application orchestration inside the allocation. Ray is the natural home for RL and post-training, where a run is a choreography of inference-heavy rollout generation feeding a smaller trainer (Chapter 1.4) rather than a single monolithic SGD loop.

SkyPilot over Slurm/cloud targets the operators whose problem is cost arbitrage and burst, not cluster ops: it abstracts a job over heterogeneous capacity (multiple clouds, multiple neoclouds, spot and on-demand) and chases the cheapest available GPUs, handling provisioning and recovery. For a tenant renting from neoclouds (Chapter 10.9) rather than running a Kubernetes estate, SkyPilot is often the right top layer. Slurm itself remains the HPC-heritage batch scheduler underneath many of these clusters and is entirely viable on its own for a homogeneous, single-tenant training estate.

Above the orchestrator sits the layer that makes training reproducible: the training control plane. It is three capabilities. Experiment tracking (the open-source and commercial lineage of MLflow, Weights & Biases, and their kin) records every run's hyperparameters, metrics, hardware, code version, and artifacts so that 'which config produced this checkpoint?' is answerable months later. Lineage and artifact management connect a shipped model back to the exact dataset snapshot, tokenizer, and base checkpoint it descended from — the same provenance discipline that the data-governance regime of Chapter 10.10 requires for legal defensibility. Sweep orchestration drives hyperparameter search: grid, random, and Bayesian/bandit methods (Hyperband, ASHA, population-based training) that adaptively kill underperforming trials to spend the GPU budget where it earns the most signal.

The hard question is where the budget goes. At frontier scale, you cannot sweep the full hyperparameter space at the target model size — a single run is the budget. The workable pattern is scaling-law-guided search: sweep cheaply at small scale, fit the scaling laws (and the maximal-update-parameterization / muP transfer that lets a learning rate found at small width transfer to large), and test the transfer assumptions before extrapolating the few choices that matter to the full run. An operator that sweeps blindly at scale burns megawatts on trials a properly designed small-scale sweep could have eliminated; an operator that skips small-scale search entirely gambles the full-run budget on untuned hyperparameters. The control plane is what makes the cheap-search-then-extrapolate workflow auditable.

Checkpointing is the mechanism that most directly converts the fleet's raw availability into goodput — the fraction of GPU-time that advances the model rather than being lost to faults and recovery. Synchronous coupling sets the stakes: SemiAnalysis reported roughly 7 days MTBF for one 512-H100 cluster at a top-tier operator. That observation is not a per-GPU scaling law; derive the named job's interruption distribution from fleet telemetry before selecting checkpoint cadence. Two costs bracket the cadence decision. Checkpoint too rarely and a failure throws away hours of progress (lost-work cost). Checkpoint too often and the write stalls (or the async drain bandwidth) eats into the step time (overhead cost). Chapter 9.4 derives checkpoint cadence from measured whole-job interruptions and checkpoint/restart cost; as whole-job MTBF shrinks, the cadence tightens.

The modern answer is asynchronous, multi-tier checkpointing, and it is a material goodput lever; 90% versus 96% is an illustrative sensitivity pair, not an industry population. Tier the writes: capture state into local memory/NVMe, commit complete shards and metadata to the intended tier, replicate to peers outside the failed node, and drain asynchronously to durable parallel-filesystem or object storage outside the relevant failure domain. Overlap the drain with computation, but measure capture pauses and bandwidth contention; GPUs can still wait. A local or peer checkpoint is eligible only if its complete committed state survives this failure. Google’s named 35K-chip TPU v5p example reports restore in less than one minute; detection, fencing, replacement allocation and the first valid step add separate clocks, as the recovery trace in Chapter 10.7 shows. The full engineering of checkpoint storage, bandwidth sizing (the assumed 14 bytes/parameter serialized-state recipe), and elastic restart lives in Chapter 9.4; the KV-cache side of the memory hierarchy is Chapter 9.7. For this chapter the requirement is narrower: your framework and orchestrator must support async/distributed/elastic checkpointing, or the most expensive part of your fleet is idle during every recovery.

Reproducibility in large-scale training is a spectrum, not a binary, and where you sit on it is a deliberate cost decision. Bit-exact reproducibility — re-run the job and get byte-identical weights — requires pinning the entire stack (framework, CUDA, cuDNN, NCCL, driver), forcing deterministic kernels (which can cost throughput; alternative reductions can change rounding order), fixing every RNG seed, and capturing the exact data order. Pay that MFU cost only where exact replay is required, and test it on the fixed stack. PyTorch documents no complete reproducibility guarantee across releases or platforms. Statistical reproducibility — re-run and get a model within noise on every eval — is a different contract, with explicit numerical and quality tolerances: it tolerates non-deterministic kernels but still requires pinned versions, captured seeds and data order, and recorded hyperparameters.

The consequence of skimping is concrete and recurring. A run that drifts because someone bumped the NCCL version mid-sweep produces results you cannot compare; a model you cannot trace back to its dataset snapshot is a model you cannot defend in a copyright dispute (Chapter 10.10) or reproduce after the person who trained it leaves. The minimum reproducibility contract for a serious operator is: pinned and recorded software versions, captured seeds, captured data-loader order, recorded full config, and immutable artifact lineage from dataset → checkpoint → eval, plus optimizer and data-loader state for resumption. That contract is cheap to maintain if the control plane enforces it from day one and ruinously expensive to reconstruct after the fact.

The five layers of this chapter form a chain, and the realized goodput of the fleet is the product of each link, not the strongest one. The framework sets the ceiling on MFU; the parallelism layout determines how close you get to that ceiling on your specific topology; the orchestrator determines how much of the wall-clock is spent training versus queued, mis-placed, or re-forming; checkpointing determines how much of a failure you lose; and reproducibility determines whether any of it is a repeatable result. A best-in-class framework at 54% MFU running on a topology-blind orchestrator that places tensor-parallel traffic across the spine, with synchronous checkpoints stalling every step, can deliver worse goodput than a simpler stack that gets every link merely competent. The operator's job is to find the weakest link and fix it, because that is the one billing megawatts for nothing.

The economic tie-back is direct. Price reliability from the named fleet's measured badput categories. A declared 90%-versus-96% sensitivity on a gigawatt training estate can be worth hundreds of millions of dollars of effective compute per year, but it is scenario arithmetic, not a sampled industry gap. That gap is won and lost in this chapter — in the parallelism layout, the gang scheduler's topology awareness, the checkpoint tier, and the discipline of pinned, traceable runs. The same goodput-vs-availability reframing that governs facility redundancy (Chapter 12.2) governs the software stack: spend on the things that protect useful training time, not on nines the synchronous workload does not value.