Node Software Stack: Drivers, CUDA/ROCm, NCCL & Firmware

Above the rack and below the scheduler sits a layer that almost no slide deck shows and almost every multi-week training failure traces back to: the node software stack. It is the vertical column of software that turns a powered, networked GPU server into a node a distributed job can actually use — the kernel driver that talks to the silicon, the CUDA or ROCm runtime the framework links against, the collective-communication library (NCCL on NVIDIA, RCCL on AMD) that carries every all-reduce, and the firmware estate (GPU VBIOS, NIC, BMC, NVSwitch, CX/Thor, PSU) that sits beneath all of it. These are not independent knobs. They are a single coupled organism with a compatibility matrix, and the central engineering fact of this chapter is that the matrix is unforgiving at scale: a synchronous training job runs at the speed of its slowest, most-skewed node, and a mismatched NCCL version across two nodes out of two thousand does not degrade the run — it hangs the collective and stalls every GPU behind it.

Most operators treat this layer as plumbing until it costs them a run. We trace the vertical stack and the driver lifecycle; the injection model (GPU Operator vs golden image) and the container-runtime layer beneath it; the AMD ROCm path as a real but taxed second source; NCCL/RCCL tuning as the difference between paper bandwidth and realized bus bandwidth; and the discipline that holds it all together — fleet-wide version pinning, the golden stack, and firmware update mechanics under attestation. The recurring fork is the same: uniformity buys goodput, per-job flexibility buys agility, you cannot maximize both, and choosing wrong shows up not as a config error but as a stalled $50M run.

Read the node stack bottom-to-top and the coupling becomes obvious. At the base is firmware — GPU VBIOS/InfoROM, NVSwitch/UALink-switch firmware, NIC firmware (ConnectX/BlueField/Thor on the NVIDIA side; Pollara/Pensando on AMD), BMC, BIOS/UEFI, PSU and CDU controllers — each with its own version, its own update path, and its own dependency on the layers above. On top of firmware sits the kernel driver (the NVIDIA data-center driver branch, or AMDGPU/amdkfd for ROCm), which must match the kernel ABI and the GPU's firmware. The driver exposes a user-space runtime: the CUDA driver API and toolkit (CUDA 13.3 as of mid-2026), or the ROCm runtime (HIP, rocBLAS, MIOpen). Above that are the math and communication libraries — cuDNN, cuBLAS, NCCL on NVIDIA; MIOpen, rocBLAS, RCCL on AMD — and finally the framework (PyTorch, JAX, vLLM) the user actually writes against.

The reason this matters is that the constraints run both ways. A given CUDA toolkit requires a minimum driver; a given NCCL build is compiled against a CUDA version; a given firmware revision is validated against a driver branch; and the framework pins a cuDNN/NCCL it was built and tested with. NVIDIA's CUDA forward-compatibility and minor-version compatibility features loosen this slightly — a newer CUDA toolkit can run on an older driver within the same major branch via the compat package — but those are escape hatches, not a license to let the fleet drift. The operative posture for a production cluster is the opposite: pin everything, identically, everywhere, and treat a deviation as a defect.

NVIDIA ships data-center drivers in two cadences, and choosing between them is the first real fork. Production Branches rotate roughly annually with about a year of full support; Long-Term Support Branches (LTSB) get a three-year lifecycle prioritizing stability over features. As of mid-2026 the R580 branch is the current LTSB, with support running to roughly August 2028, paired against CUDA 13.x. A fleet that values determinism over feature velocity pins to an LTSB and upgrades on a deliberate cadence; a fleet chasing the newest accelerator generation or a kernel feature may have to ride a Production Branch and accept the shorter support window. The consequence of choosing wrong is asymmetric: ride too-new a branch and you inherit regressions on hardware you cannot afford to debug under a live run; ride too-old a branch and you cannot enable the next density generation when the ramp arrives.

The second fork is how the driver lands on the node. Two models dominate, and they trade agility against bare-metal determinism.

The GPU Operator (NVIDIA's, and AMD's equivalent) is the Kubernetes-native answer: it deploys and lifecycles the containerized driver, the container toolkit, the Kubernetes device plugin, DCGM/DCGM-exporter, the MIG manager, and node-feature discovery as a coordinated set, and it performs rolling driver upgrades by cordoning and draining a node before swapping its stack. That is exactly the right tool for a cloud-native, inference-heavy, or heterogeneous fleet where you want to roll a driver across thousands of nodes without re-imaging. The cost is more moving parts in the critical path and a dependency on the container runtime being correctly wired (the NVIDIA Container Toolkit injecting /dev/nvidia*, libraries, and the right capabilities into the container; the equivalent CDI-based device injection on the AMD side). For change-controlled Slurm/HPC training fleets the opposite instinct usually wins: bake the driver and toolkit into a golden OS image, attest it at boot, and treat every node as a frozen, identical artifact — the determinism that synchronous training rewards. Many large operators run a hybrid: host-baked driver for determinism, Operator-managed plugins/DCGM/MIG for the layers that benefit from declarative lifecycle. The provisioning machinery that produces and promotes those images is the subject of Chapter 10.5.

For most of the last decade the node stack was a single-vendor question because the alternative was not credible at production scale. That has changed. ROCm 7.x (7.0 shipped September 2025, support for the MI350X/MI355X; the line has since moved to 7.1/7.2) is the first AMD software stack that a serious operator can stand up a multi-thousand-GPU training or inference fleet on without heroics. The strategic prize is real: a credible second source breaks single-vendor allocation pain and lands roughly 15-30% lower hardware cost per unit of compute, plus negotiating leverage that is worth more than the sticker discount. The reason it is still a fork and not a free lunch is the ROCm tax — the engineering cost of the remaining ecosystem gaps.

Two things make ROCm viable today. First, RCCL maintains NCCL-API parity: it implements the same collective API surface NCCL exposes, so frameworks that call NCCL can call RCCL with minimal change, and the tuning concepts (algorithm/protocol selection, rail awareness, LL/LL128 protocols) carry over. Second, the framework story has matured — upstream PyTorch and JAX run on ROCm, and inference engines like vLLM support AMD targets. Where the tax still bites is in the long tail: custom CUDA kernels and Triton paths that assume NVIDIA intrinsics, profiling and debugging tooling that is less mature than Nsight/CUPTI, library coverage gaps (a kernel that exists tuned in cuDNN but not yet in MIOpen), and — the number that decides TCO — the realized-MFU gap. Independent benchmarking has shown AMD parts delivering a meaningful but workload-dependent fraction of NVIDIA's realized throughput on identical models; on some inference workloads the gap has closed substantially, on others it persists. The node-stack consequence is concrete: a dual-vendor fleet means two golden stacks, two firmware estates, two sets of NCCL/RCCL tuning tables, and two on-call runbooks. You buy second-source leverage and pay in operational surface area. The deeper hardware and TCO treatment of this choice lives in Chapter 7.3; the lock-in economics of CUDA vs ROCm vs XLA vs Neuron in Chapter 7.9.

The collective library is the most performance-critical and most tuning-sensitive component in the node stack, because at training scale above a few hundred GPUs the fabric — not the GPU — sets job completion time, and NCCL/RCCL is what drives the fabric. Its job is to execute the collectives that define distributed training (all-reduce, all-gather, reduce-scatter, all-to-all) at the highest possible bus bandwidth (busbw) — the effective throughput a collective achieves relative to the link's theoretical ceiling. The acceptance bar operators gate handoff on is concrete: NCCL all_reduce_perf should reach roughly 92% of theoretical fabric bandwidth scaling from two nodes to the full cluster — about 370 GB/s busbw on a 400G (NDR) fabric. Falling short of that is not a benign inefficiency; it is a tax amortized across the entire run, because every GPU waits on every collective.

NCCL auto-tunes well out of the box, and for many fleets the right answer is to ship defaults and not touch the knobs. But the last 10-20% of busbw on a specific topology is recovered only by tuning the library to the fabric you actually built: algorithm selection (ring vs tree vs the newer adaptive trees, CollNet/PAT), protocol selection (Simple/LL/LL128), rail awareness so traffic stays on the rail-optimized topology, channel and buffer sizing, and — where the fabric supports it — SHARP in-network reduction. NCCL 2.27+ composes NVLink-SHARP and IB-SHARP, cutting the GPU SM count consumed by a reduction from ~16 to ≤6 and halving the data on the wire; the 2.28+ line adds symmetric-memory kernels and Multimem multicast over NVLink-SHARP within an NVL72 domain. The in-network-compute mechanics live in Chapter 8.6, and the topology/oversubscription decisions that NCCL tuning must match in Chapter 8.5. The choice is sharp: ship defaults and accept ~80-90% of achievable busbw, or invest in per-topology tuning and recover the rest. Either way the tuning tables become part of the pinned golden stack, because a tuning skew is a silent bandwidth skew across nodes.

Everything above converges on one operating model: the golden stack — a single, named, version-pinned tuple of driver + CUDA/ROCm + cuDNN/MIOpen + NCCL/RCCL + container toolkit + firmware revisions, validated together and promoted as one atomic artifact across the entire fleet. The golden stack is the fleet's answer to the compatibility matrix: instead of reasoning about whether this CUDA works with that driver on those nodes, you validate one combination, hash it, and refuse to run anything else. Its three properties are non-negotiable. It is uniform — byte-identical on every node, enforced by an admission gate, because synchronous training cannot tolerate skew. It is versioned and attested — every node can prove which stack it booted, so drift is detectable and a non-conforming node is automatically drained. And it is promoted as a unit through stages (lab → canary node pool → ring rollout → fleet), never component-by-component in production, because a component upgrade that is fine in isolation can be a busbw or hang regression in combination.

The cost of the golden-stack discipline is agility: you cannot let a single team try a newer NCCL on a shared training cluster without re-pinning and re-validating the whole fleet, and that friction is real. The cost of not having it is worse and recurs forever: version skew hangs, silent bandwidth regressions, irreproducible runs, and a fleet where 'works on my node' is a daily occurrence. The right posture for a production training cluster is unambiguous — one golden stack, pinned, attested, promoted in rings, with per-job overrides confined to isolated dev/experiment pools that never share a synchronous collective with the production fleet.

Firmware is the part of the node stack operators most often defer — it is tedious, it requires reboots, and it rarely breaks loudly. That deferral is exactly the trap. The GB200 ramp made the point at scale: integration reliability hinged on firmware, with NVLink copper-backplane issues and NVL36×2 cross-rack signal-integrity problems fixed partly through firmware revisions. A node carrying stale NIC, NVSwitch, or VBIOS firmware is a node that contributes gray failures — link flaps, lane dropouts, intermittent throttling — that do not crash the node but do drag synchronous collectives and accumulate into the every-few-hours interruption rate that destroys multi-week run economics. Firmware is the slow-moving layer that quietly decides reliability.

The modern firmware estate is managed out-of-band and as code. The OCP GPU Firmware Update specification standardizes the mechanics — Redfish for the management API, PLDM-over-MCTP for the update transport, and secure out-of-band update so firmware can be staged and applied without a host agent. At fleet scale, firmware updates follow the same ring discipline as the golden stack: rolling/canary rollout, drift detection against a pinned firmware baseline, a dependency matrix that sequences firmware against driver/CUDA so you never strand a node between incompatible layers, and rollback when a canary regresses. Two constraints make this harder than ordinary patching: every update is a scheduling event (a node must drain a job before it can reboot), and firmware is a high-value attack surface — so updates must be signed and attested against a hardware root of trust (Caliptra/DICE, measured boot), the subject of Chapter 11.4. The day-2 firmware-estate operations, change-management, and refresh mechanics are deepened in Chapter 14.9.

One layer is easy to overlook because it is invisible when it works: the container runtime that actually exposes GPUs to a workload. On NVIDIA, the NVIDIA Container Toolkit hooks the OCI runtime to inject the GPU devices (/dev/nvidia*), the driver user-space libraries, and the right capabilities into the container at start — so the containerized framework links against the host's pinned driver rather than a stale library baked into the image. The industry is migrating this device-exposure model to the vendor-neutral Container Device Interface (CDI), which both NVIDIA and AMD implement, decoupling device injection from any single runtime. The consequence for the golden stack is subtle but important: even in a containerized fleet, the driver is a host property and the container inherits it, which is why a host-baked driver plus Operator-managed user-space is a coherent posture rather than a contradiction. Kubernetes is also moving the resource-request model itself — from counting whole GPUs toward Dynamic Resource Allocation (DRA), which lets a job claim attributes (MIG geometry, NVLink topology, memory) rather than a count — which pushes more of the node-stack's capabilities (partitioning, topology) into the scheduling layer treated in Chapter 10.2 and Chapter 10.3.

The node-stack failures that recur are all variations on one mistake — treating a coupled organism as a set of independent knobs:

• Letting NCCL drift across nodes. The textbook self-inflicted goodput wound: a node re-imaged with a slightly different NCCL joins a collective and hangs the run, or worse, completes at degraded busbw for weeks. Prevent it with an admission gate on the golden-stack hash, not with a wiki page asking people to be careful.
• Upgrading one component on a live training fleet. A 'safe' point-upgrade to a driver or NCCL that passes in isolation but regresses in combination, now mixed into production runs. Promote the stack atomically through rings, never component-by-component.
• Deferring firmware until something breaks. Stale NIC/NVSwitch/VBIOS firmware contributes gray failures — link flaps, throttling, lane dropouts — that never crash a node but drag every synchronous collective and inflate the interruption rate. Firmware is a scheduled, attested, ring-rolled estate, not an emergency activity.
• Going dual-vendor for the discount without budgeting the surface area. A second vendor doubles golden stacks, firmware matrices, and on-call rotations. The 15-30% hardware saving is real, but so is the operational cost — go dual-vendor for leverage and scale, not for a line-item discount on a small fleet. → Chapter 7.3.