Post-Training, Fine-Tuning & RL: The Hybrid Middle

Of the five workload archetypes in Chapter 1.1, post-training is the one operators most reliably mis-scope, because the word hides a contradiction. "Post-training" sounds like training, a gradient-descent job that updates weights, so it gets specced like training: dense racks, a non-blocking fabric, the works. But for rollout-heavy reasoning and agentic RL, generating the data the gradient step learns from can dominate elapsed training time, and that generation is autoregressive inference. Supervised fine-tuning and offline preference optimization have a different profile, so the post-training label alone cannot size a generation fleet. Get the balance right and you build a cheap, flexible, high-goodput cluster. Get it wrong and you pay for a trainer's fabric to run an inference fleet, or you starve the learner on an inference cluster that has no trainer.

This chapter engineers that hybrid middle. It maps the post-training landscape: supervised fine-tuning (SFT), preference optimization (RLHF/RLAIF/DPO) and large-scale reinforcement learning for reasoning (PPO, GRPO and relatives). It derives when RL is inference-heavy training from the token economics of rollout generation, then walks the two infrastructure forks that follow: collocated vs disaggregated execution and the staleness budget that governs asynchronous designs. It sizes the fine-tuning end of the spectrum, LoRA/QLoRA vs full fine-tune, and ends with capacity planning for spiky, project-based demand, where the choice is dedicated capacity versus time-shared tenancy on someone else's fabric.

Post-training is everything that happens to a model after the pre-training run produces a base model. It spans a spectrum from cheap-and-bounded to expensive-and-open-ended, and the infrastructure profile changes radically across it. A facility scoped for one end serves the other badly.

Supervised fine-tuning (SFT) is the closest to classic training: a labeled dataset of prompt/response pairs drives ordinary gradient descent. It is small (often thousands to low-millions of examples) and bursty (it starts and finishes inside hours to days). It is tightly coupled only at the scale of the model being tuned, which for most fine-tunes is a single node or a small handful, not a thousand-GPU supercomputer. SFT is the cheapest post-training to host and the easiest to time-share.

Preference optimization (RLHF with human feedback, RLAIF with AI feedback, and simpler offline variants like DPO) aligns a model to preferences over outputs. Classic PPO-style RLHF is the heavyweight: it runs up to four models concurrently (a policy/actor, a frozen reference, a reward model and a value/critic), so four 70B-class models at two bytes per weight are 560 GB of weights alone (4 × 70 × 10⁹ × 2 bytes). Device placement, optimizer state, activations and runtime headroom need separate budgets. Offline DPO uses a contrastive loss over chosen and rejected responses without a reward-model fleet or policy sampling during optimization. It evaluates both responses under the trainable policy and needs their frozen-reference log-probabilities, computed online or cached before training. Budget the paired token workload and reference evaluation explicitly; its infrastructure remains in the supervised-fine-tuning class. The choice between PPO-style and DPO-style alignment is therefore as much an infrastructure decision as an algorithmic one.

Large-scale RL for reasoning (the GRPO/PPO family behind the 2025–2026 wave of reasoning models) is the archetype that breaks the "post-training = training" intuition. It alternates two phases with opposite hardware profiles: a rollout phase that samples long trajectories from the current policy (pure inference) and a policy-update phase that runs a comparatively small synchronous gradient step. In rollout-heavy configurations, generation dominates elapsed time, and that reorganizes the cluster. Chapter 10.8 covers the frameworks, orchestration and checkpointing an RL stack runs on; this guide does not derive the GRPO/PPO algorithms themselves.

Here is the derivation, because the conclusion is counterintuitive and the capital at stake is large. In RL for reasoning, the model improves by trying things and being rewarded. Each training step requires the current policy to generate candidate responses (rollouts), which are then scored and used to compute a gradient. For reasoning and agentic tasks those rollouts are long: chains of thought, tool calls and multi-turn trajectories. AgentMath reports trajectories averaging about 24K tokens with approximately 27 tool invocations. Size generation and tool-service demand from your workload’s measured length distribution. You also generate many rollouts per prompt (GRPO samples a group of completions per prompt to estimate a baseline). Generation is autoregressive decode, one token at a time, and decode is limited by HBM bandwidth when weight and KV reads saturate it. The gradient update that follows takes a fraction of the wall-clock.

The numbers are stark, but they come from specific setups. In AgentMath’s conventional Qwen3-8B batch-synchronous baseline, with long sequences and frequent tool calls, rollout generation takes 70–80% of training time, including rollout waiting. In SortedRL's motivating example, rollouts occupy about 70% of elapsed training time at a 16K maximum generation length. Both are phase-duration shares in one setup each, not a universal RL compute, FLOP or energy share. The rollout engines repeatedly stream weights and KV-cache through memory exactly as an inference server does. For these generation-heavy traces, rollout service governs much of the elapsed step. That duration does not give the trainer-to-rollout GPU ratio. Any scoping that treats the whole cluster as a synchronous trainer risks paying for a non-blocking fabric and uniform max-density racks without measuring generation GPU-seconds, learning work and their overlap.

Once you accept that an RL cluster is two workloads, a rollout fleet and a trainer, the central architectural question is whether they live in the same GPU pool or separate ones. This fork sets utilization, fabric and the failure modes you will fight.

Collocated designs put rollout and training on one shared resource pool and alternate phases in time: the GPUs generate, then switch to updating, then generate again. The win is simplicity. The frameworks that pioneered this (the verl family and relatives) make it the default for moderate scale. Both phases reuse the same GPUs. Utilization depends on occupancy within each phase, long-tail rollout waiting, environment delays and the time spent resharding or offloading and reloading state. Generation and updates run in series, and the pool must fit the more demanding phase. At large scale that serial execution is the wall-clock the rollout share warns about.

Disaggregated designs place a dedicated rollout/generation fleet and a separate trainer on different GPU pools, connected by a weight-sync path. The 2025–2026 systems literature has converged toward this architecture (StreamRL, AReaL, ROLL Flash, ROLLART and kin) because it lets each side use best-fit hardware: decode-heavy rollouts on bandwidth-optimized inference GPUs, the policy update on compute-optimized trainer GPUs, and environment and reward computation on CPU clusters or serverless. Asynchronous execution can overlap the phases; keeping both pools busy requires balanced stage throughput and timely weight delivery. Separate pools can still hold a synchronous barrier. What makes disaggregation pay is overlapping generation and training, and overlap means the trainer consumes rollouts from a slightly older policy, the staleness problem treated next. Placement and schedule are two decisions, so score idle time, memory pressure, weight-transfer cost, policy lag and convergence at a fixed quality target rather than assuming the pair moves together. Disaggregation also demands a deliberate weight-broadcast path and a fabric that carries fresh policy weights to the rollout fleet fast enough to keep it from drifting too far off-policy.

Asynchrony is where disaggregated RL buys its speed, and where it can lose its convergence. When the rollout fleet and the trainer run in parallel, the trajectories the trainer learns from were generated by a policy that is now a few updates old. That gap is staleness (or policy lag), and it is the governing tuning knob in an async RL system. Allow zero staleness and you are back to synchronous, on-policy training: stable, but the trainer idles while rollouts generate and the rollout fleet idles while the trainer steps. Allow unbounded staleness and the trajectories become so off-policy that the gradient points the wrong way; training destabilizes, with the gradient-norm spikes and collapses the literature documents.

The 2026 engineering answer is a bounded staleness budget: let the trainer consume rollouts up to k policy versions old, no more, paired with off-policy corrections (importance-sampling variants, variance control) that keep the stale gradient honest. Well tuned, the payoff is large: recent variance-controlled async methods match the best synchronous accuracy on long-context agentic RL roughly 2.5x faster in wall-clock (about 42 hours versus 105 hours in one published comparison) and keep improving past the synchronous ceiling. The staleness bound doubles as an infrastructure parameter: it sets how often you must broadcast fresh weights across the fabric to the rollout fleet, and so how much weight-sync bandwidth you must provision. Tighter staleness means more frequent broadcasts, which means more east-west traffic on the sync path. The GOODPUT lens applies to RL too: the cluster's useful output is gradient steps that actually improve the policy, and both an idle trainer (too synchronous) and a divergent one (too stale) are goodput you paid for and did not get.

At the fine-tuning end of the spectrum, the decisive fork is parameter-efficient fine-tuning (PEFT) versus full-weight fine-tuning, and the capacity difference is large. A full fine-tune updates every weight and budgets weights, gradients, optimizer state and activations. For the reference mixed-precision Adam layout in Chapter 10.8, two-byte weights and gradients plus an FP32 master copy and two FP32 moments total 16 bytes per parameter before activations, temporary buffers and allocator headroom. The master copy and moments account for 12 of those bytes. Guide calculation: a 70B model therefore has 1.12 TB of persistent state in this layout; sharding and offload determine its per-device residency and hardware allocation. LoRA freezes the base weights and trains a small low-rank delta, cutting the trainable parameter count by orders of magnitude (the share depends on rank, targeted matrices and model architecture; count your recipe’s trainable parameters) and the optimizer-state memory with it. QLoRA stores the frozen base in 4-bit form and trains adapters, while computation still needs higher-precision dequantized tensors, activations, input gradients and adapter optimizer state. Reserve the measured peak for the intended sequence length, microbatch and checkpointing recipe, including host memory and transfer capacity when optimizer paging is used.

The consequence for capacity planning is concrete. QLoRA's headline result is fine-tuning a 65B-parameter model on a single 48 GB GPU, collapsing a requirement from >780 GB of GPU memory to <48 GB without degrading task quality. That demonstrates that the paper’s 65B fine-tuning recipe fits one 48 GB GPU; test your intended recipe and its quality target before booking shared capacity. Where it holds, fine-tuning stops being a capacity problem and becomes a scheduling problem: LoRA/QLoRA jobs are the small, interruptible tenants you backfill onto an inference fleet's idle troughs or a training cluster's gaps. The decision rule: if the fine-tune can be expressed as a low-rank adapter and quality holds, don't reserve dedicated training capacity for it; fit it onto capacity you already own. Reserve full-fine-tune-class allocations only when adapters demonstrably do not reach the quality bar.

Post-training demand does not behave like training or inference, and that is the planning problem. A pre-training run is a forecastable monolith: you know the GPU-months before you start. An online-inference fleet has a diurnal, statistically stable load you can size to a percentile. Post-training is neither. It is spiky and project-based. An alignment project, a new reasoning RL run or a customer fine-tune each spins up, consumes a burst of heterogeneous capacity for days or weeks, then ends. The aggregate demand is a sum of poorly correlated bursts that no single reservation curve fits well. The open capacity-planning question is dedicated clusters versus time-shared fabric.

Dedicated post-training capacity buys predictability and isolation (your RL run is not preempted by someone else's inference spike) at the cost of low average utilization, because spiky demand leaves the pool idle between projects. It is the right call only when post-training is a continuous, first-class product line, such as a frontier lab iterating on reasoning models, rather than an occasional activity. Time-shared tenancy, running post-training on slack from a training or inference fabric, inverts the tradeoff: high utilization because you backfill troughs, at the cost of contention and the engineering to preempt cleanly. The fit is natural: RL rollouts are inference, so they backfill an inference fleet's idle capacity with the same engines, and LoRA/QLoRA fine-tunes are small and interruptible, so they slot into either fabric's gaps. The heterogeneity that makes RL hard to scope on a single fabric is what makes it a good time-sharing citizen across two.

Post-training does not escape the density-and-cooling physics of the other archetypes. It inherits a split version of it. The trainer pool inherits its accelerator's heat split, airflow and inlet limits, heat flux and facility-water requirements. Current NVL72-class trainers require direct-to-chip liquid because of that product design, not because of a universal rack-kW threshold. Test the rollout pool independently: an inference label does not choose air or RDHx; the specific server's heat split, airflow and inlet limits, water interface, residual-room duty and service conditions do. The retrofit hazard from Chapter 1.1 applies unchanged: if the supported trainer roadmap may require liquid capture, price the structural, distribution, isolation, CDU and heat-rejection reserve at scoping time, because retrofitting those missing interfaces can strand capacity. A post-training facility is rarely uniform: separate trainer and rollout zones can need different interfaces. Check floor loading, water and electrical headroom against actual placement; only an intended interchange zone needs to support the denser configuration. Chapter 5.1 and Chapter 5.4 engineer the cooling-selection limits.