Orchestration Architecture & the Scheduling Plane

You can build a flawless data center — perfect PUE, a non-blocking fabric, golden-image nodes — and still waste paid GPU-hours because the layer that places work on the hardware is mis-designed. The orchestration stack is where megawatts of energized, depreciating silicon become either useful tokens or idle heat. It is the most software-defined decision in the facility and, perversely, the one most often inherited by accident: the HPC team brings Slurm because that is what they know, the platform team brings Kubernetes because that is what the rest of the company runs, and the two never reconcile until a frontier training run and an inference fleet are fighting over the same GPUs through two schedulers that cannot see each other.

The architecture of that layer breaks into three planes every fleet must design coherently — the workload scheduler, the node runtime, and the fleet/control plane. Then come the choices themselves: Slurm versus Kubernetes, the batch schedulers that supply queue policy and group scheduling on K8s (Volcano, KAI, Run:ai), the Slinky hybrids that try to run both at once, and Dynamic Resource Allocation (DRA), the substrate change that finally lets Kubernetes describe a GPU as something richer than an opaque countable unit. Each is scored by what it costs downstream in goodput, utilization, operational headcount, and lock-in. The deeper mechanics — topology-aware placement, multi-tenant isolation, the node software stack, autonomous recovery — each get their own chapter in Part 10; this one is the map that tells you which plane owns which problem. Assign each device one allocation owner: two schedulers may share inventory visibility, but may not independently grant the same GPU.

A working AI fleet runs three loosely-coupled control loops, each operating at a different timescale and owned, ideally, by a different on-call rotation. Conflate them and you get the recurring failure mode of GPU operations: a node-health problem that looks like a scheduler bug, a fabric event that looks like a job failure, a driver mismatch that surfaces as a mysterious collective hang three layers up.

• The workload-scheduling plane answers which job runs on which GPUs, when. It owns queues, priority, fair-share, gang/co-scheduling, preemption, and backfill. This is Slurm, or Kubernetes-plus-a-batch-scheduler. It operates at the timescale of seconds-to-minutes and is the plane this chapter is principally about.
• The node runtime plane answers can this node actually run the work correctly. It owns the GPU driver, CUDA/ROCm, NCCL/RCCL, firmware, the container runtime, and the device exposure (device plugin or DRA driver). It operates at the timescale of a node's lifecycle and is treated canonically in Chapter 10.4. The scheduler trusts this plane; when the trust is misplaced — a single node with a mismatched NCCL or a degrading NVLink — the scheduler happily places a 1,024-GPU job onto the rotten node and the whole gang hangs.
• The fleet / control plane answers is the fleet healthy, and how do we keep it that way. It owns provisioning and bring-up (Chapter 10.5), observability and health telemetry (Chapter 10.6), and autonomous fault detection and recovery (Chapter 10.7). It operates at the timescale of hours-to-days and is where products like NVIDIA Mission Control and Base Command Manager live.

The seam that matters most is between the workload plane and the fleet plane, and it is mediated by a single verb: cordon-and-drain. When the fleet plane detects a sick node — via DCGM, an XID error, a fabric counter — it must be able to tell the workload plane to stop scheduling onto it and to migrate or checkpoint the work that is already there, without a human in the loop. Meta's named Llama 3 405B run on 16,384 H100s logged 419 unplanned interruptions over 54 days — about one every three hours for that job — while automation limited manual intervention. Treat that as an observed job cadence, not a per-GPU MTBF or scale law; the scheduler and recovery SLO must be sized from the project's measured whole-job event definition and distribution.

Two schedulers dominate the workload plane, and they were born to solve different problems. Slurm comes from the supercomputing world: it assumes a long-lived, tightly-coupled batch job that wants all of its nodes at once, communicates over a fast fabric, runs to completion or fails as a unit, and is submitted by a researcher who thinks in sbatch scripts and partitions. Kubernetes comes from the web-services world: it assumes a fleet of independent, restartable, horizontally-scalable containers behind a load balancer, declaratively reconciled toward a desired state, with rolling upgrades and self-healing as first-class verbs. Neither was designed for the other's workload, and the friction at the boundary is the central operational problem of AI orchestration in 2026.

A widely repeated trade-press split puts roughly 70% of AI/HPC scheduling on Slurm, ~20% on Kubernetes, ~10% in-house. Read it as a direction, not a market share: no published sampling frame says whether it counts organizations, clusters, installed GPUs, or job-hours. Training — synchronous, gang-scheduled, fabric-bound — gravitates to Slurm because gang scheduling is native: sbatch -N 256 gets you all 256 nodes simultaneously or the job waits, which is exactly the all-or-nothing semantics a synchronous all-reduce job needs. Inference — bursty, loosely-coupled, autoscaled, behind a service mesh — gravitates to Kubernetes because that is what Kubernetes was built for. One operator observation points the same way: roughly 90% of GPU-cloud customers use Kubernetes for inference while ~50% use Slurm for training — a SemiAnalysis customer observation relayed by CoreWeave in April 2025, with that sample's limits, not a census. The two workloads pull toward opposite schedulers, which is why so many fleets end up running both.

The architecture question is whether your fleet's workload mix justifies one plane or two, and if two, whether you federate them or converge them. That choice carries the real downstream cost: headcount, stranded capacity at the boundary, and the blast radius of a control-plane outage.

Read the table plainly: Slurm fits an HPC batch operating model, Kubernetes fits an existing container service platform, and a hybrid buys access to both at the price of complexity you must staff for. Test those fits against your workload and controller-failure cases. The common mistake is choosing the hybrid by default — "we'll run both, best of both worlds" — without budgeting for the fact that you now have two schedulers, two notions of a node, two quota systems, two failure surfaces, and two on-call rotations that must agree on what "this node is unhealthy" means. Convergence tooling can coordinate the handoff, but convergence is not free; it is a deliberate investment justified only when the workload mix truly demands both planes on the same physical fleet.

Vanilla Kubernetes schedules pods independently — which is fatal for a distributed training job. If a 64-pod job gets 60 pods placed and then blocks waiting on the last 4, those 60 GPUs sit idle holding the gang hostage, and two such half-placed jobs can deadlock each other indefinitely. This is the partial-gang / resource-deadlock problem, and the K8s batch-scheduler ecosystem exists to solve it. Each of the three main options solves it with gang (co-)scheduling — place all pods of a job atomically or none — and then layers AI-specific scheduling on top.

• Volcano is the CNCF-incubating batch system for Kubernetes — the de facto open-source default. It brings gang scheduling, queue/fair-share, preemption, backfill, and topology-aware plugins, and integrates with the training operators (PyTorch, MPI, Ray). It is the safe, vendor-neutral choice and the one you reach for if you want to avoid lock-in.
• KAI Scheduler is NVIDIA's Kubernetes-native AI scheduler, open-sourced under Apache-2.0 in April 2025 — the engine extracted from the Run:ai acquisition. It adds gang scheduling, hierarchical fair-share quotas, bin-packing and spread strategies, GPU sharing/fractioning, and native DRA and topology-awareness. Open-sourcing it pushed these capabilities toward a commodity baseline.
• Run:ai is the commercial platform (now NVIDIA, bundled into NVIDIA AI Enterprise) layered on KAI: fractional GPU, policy/quota governance, a multi-tenant UI, and integration with the broader NVIDIA fleet stack. You buy it when you want the managed product and the support contract, not the assembly job.

The decision here is a build-vs-buy in miniature. Volcano is the open, neutral baseline you operate yourself. KAI is the open NVIDIA-blessed engine — more AI-specific, but you are now standing on NVIDIA's roadmap. Run:ai is the supported product on top of KAI. Picking KAI or Run:ai buys you fractional-GPU and quota machinery out of the box (relevant the moment you have multi-tenancy — Chapter 10.3); picking Volcano keeps you maximally portable across vendors and avoids deepening NVIDIA platform dependence at the orchestration layer.

For a decade, Kubernetes treated a GPU as a countable integer: nvidia.com/gpu: 8, an opaque unit, all alike, node-local. That model breaks the moment GPUs stop being fungible — when you care which GPUs (two on the same NVLink island), in what shape (a specific MIG geometry), with what attributes (≥40 GB free, a particular topology), or shared across pods. Dynamic Resource Allocation (DRA) replaces counting with claiming: a workload describes the properties of the devices it needs via a ResourceClaim, and a vendor-supplied DRA driver plus the scheduler resolve the actual allocation. It is the same conceptual move as PersistentVolumeClaims for storage, applied to accelerators.

The timing is the news. DRA graduated to GA in Kubernetes 1.34 (27 August 2025) — the core resource.k8s.io APIs are now stable; partitionable devices and consumable capacity reached beta in Kubernetes 1.36, while MIG, time-slicing, and fractional-GPU support remains driver-specific — and downstream distributions followed (DRA GA in OpenShift 4.21, March 2026). It changes what the scheduling plane can express natively: attribute-based selection ("2 GPUs on the same node interconnected by NVLink, min 40 GB VRAM"), driver-backed fine-grained partitioning (MIG, time-slicing, fractional GPU) without static 1:1 pod-to-GPU mapping, and network-attached devices. Vendor reports cite materially higher utilization under DRA than under static device plugins, through better packing and sharing — a direct goodput-and-economics lever.

Whether to adopt DRA now or run one more cycle on device-plugin + custom-scheduler comes down to ecosystem maturity. DRA is GA, but the surroundings — driver maturity across vendors, scheduler-plugin integration, operational tooling — are still settling in mid-2026. For a greenfield K8s cluster being stood up today, DRA is the forward-looking bet and the model the topology-aware work in Chapter 10.2 increasingly assumes. For an existing production fleet with a working device-plugin setup, the conservative path is to pilot DRA on a partition while keeping the proven model in production until the driver/tooling story for your specific hardware is boring. Betting a brand-new revenue cluster on a still-stabilizing path is the kind of avoidable risk that shows up as goodput loss in month one.

If you genuinely need both Slurm's batch semantics and Kubernetes' ecosystem on the same physical fleet, the answer the industry converged on is Slinky — the open-source project (from SchedMD, now NVIDIA) that bridges the two rather than forcing a choice. It has two complementary pieces. The slurm-operator runs Slurm on Kubernetes: Slurm controller and compute daemons as managed K8s workloads, so you get Slurm's batch/gang semantics with Kubernetes lifecycle management underneath. The slurm-bridge goes the other way: it lets Slurm act as a scheduler for Kubernetes, so Slurm's allocation logic governs placement of K8s-submitted work. NVIDIA’s 9 April 2026 report describes Slinky slurm-operator production clusters above 8,000 GPUs running LLM training and multinode inference; qualify the actual hybrid before transferring that scale result.

The promise is one physical fleet, one pool of GPUs, both workload styles — no more carving the cluster into a static "Slurm half" and "K8s half" that strand capacity at the boundary. In exchange, you are now operating, patching, and debugging two scheduling systems wired together, with two topology models, two quota regimes, and a non-trivial integration surface where the interesting failures live. Convergence pays off for operators whose workload mix is genuinely bimodal and whose utilization losses from static partitioning exceed the operational cost of the hybrid. For a fleet that is ~90% training or ~90% inference, the simpler answer — pick the one scheduler that fits the dominant workload and serve the minority case with a small dedicated partition — is usually the better engineering call.

Stitched together, a 2026-current reference fleet looks like this — read it bottom-up, as planes stacked by timescale. At the base, bare metal is provisioned as code: BMC/Redfish bring-up, PXE, golden images, and a join step that registers each node with the control plane (Chapter 10.5). On every node sits the node runtime: a pinned, identical stack of GPU driver, CUDA/ROCm, NCCL/RCCL, and firmware, exposed to the scheduler through the GPU Operator's device plugin or a DRA driver (Chapter 10.4). The hard rule lives here — every node in a synchronous job runs the same tested, attested stack tuple of driver, CUDA, cuDNN, and NCCL, because an unqualified combination is what produces collective hangs, silent bandwidth degradation, or wrong results.

Above the nodes sits the workload-scheduling plane — Slurm, or K8s + Volcano/KAI, or both via Slinky — which must be topology-aware: it has to know that the 72 GPUs in an NVL72-class rack (GB300 NVL72 today, GB200 NVL72 before it, VR200 NVL72 ramping since its August-2026 production start) form one coherent NVLink domain and schedule the rack as a block, not as 72 interchangeable units (Chapter 10.2). Wrapping everything is the fleet/control plane: DCGM/NVML telemetry and XID/SXID parsing feed an observability stack (Chapter 10.6); a health-and-recovery loop cordons sick nodes, drains their work, swaps in hot spares, and triggers break-fix without a human (Chapter 10.7); and an IaC/GitOps layer keeps the whole thing declaratively reconciled. The architectural test of this topology is whether a node can fail and the fleet detect, drain, recover, and resume — autonomously — fast enough to meet the named workload’s goodput requirement. Meta’s reported >90% effective training time and roughly three-hour interruption cadence describe its Llama 3 campaign, not this fleet’s acceptance threshold. If manual recovery misses the required service deadline, automate or reserve the capacity needed to close that measured gap, whichever scheduler you picked.

The last fork in this chapter is whether to assemble the orchestration platform from open-source parts or buy a managed product. The assemble path — Slurm or K8s, plus Volcano/KAI, plus the GPU Operator, plus a Prometheus/Grafana/DCGM observability stack, plus your own IaC and recovery automation — gives maximal control and no per-node licensing, at the cost of a standing platform-engineering team that owns the integration, the upgrades, and the 3 a.m. pages when a driver bump breaks a collective. The buy path — NVIDIA Mission Control / Base Command Manager, a hyperscaler's managed Slurm/K8s, or a neocloud's turnkey cluster — compresses time-to-first-job and hands you autonomous recovery and validated golden stacks out of the box, at the price of a license, a vendor's roadmap, and less ability to customize the deep internals.

The deciding variables are the same three that govern most build-vs-buy calls in this guide: scale (a large, durable fleet amortizes a platform team; a small or short-lived one cannot), differentiation (if orchestration is a competitive edge — a neocloud selling goodput — you build; if it is undifferentiated plumbing, you buy), and time-to-power (buying collapses months of integration into days). The recurring mistake is the mid-size operator who builds a bespoke platform to save license fees, then spends three senior engineers' salaries maintaining it — burning far more than the license, and shipping lower goodput than the managed product would have, because they are re-deriving autonomous recovery that a vendor already shipped. The productization view of this same fork — what operators actually sell on top of these platforms — is the subject of Chapter 10.9.