Software, Orchestration & Service Delivery

The scheduler decides whether your GPUs run jobs or sit idle; choosing Slurm, Kubernetes, or both is a bet on your workload mix and the cost of operating two control planes.

On rack-scale machines placement is a performance contract: a job inside one NVLink domain runs at full bandwidth; split across the boundary, it falls off a bandwidth cliff no tuning recovers.

Sharing a GPU means choosing performance isolation (whole-GPU, MIG, MPS, time-slicing, fractional) and security isolation (process, container, VM, confidential VM) separately; conflating them sells a partition as a boundary.

Driver, CUDA/ROCm, NCCL/RCCL, and firmware form one versioned organism, pinned identically across every node; in a synchronous cluster a one-line version skew hangs the collective rather than slowing it.

Declarative, pipeline-driven provisioning sets how many GPU-hours a fleet burns between powered-on and first paid job — at initial bring-up, at every refresh, and at every node failure.

GPU-fleet observability exists to protect goodput: what you detect, how fast you attribute it, and what telemetry you can afford to keep decide whether the cluster trains or restarts.

Job-level MTBF collapses with node count, so training reliability is a control-plane problem: detect the fault, eject the node, and restart from a recent checkpoint in minutes, or lose the goodput.

Framework choice, parallelism layout, orchestration, and checkpointing set how much of a fleet's paid arithmetic becomes trained weights; a wrong layout can idle half the FLOPS at full price.

A cluster becomes a product when customers can buy it, run a job, be metered, and leave; the rung you sell on the bare-metal-to-serverless ladder sets isolation, SLA exposure, billing, and margin.

The legal and privacy posture of your data is an architecture decision: made wrong, it can force a retrain, geo-fence a hall, or hand twenty million customer conversations to opposing counsel.

Inference serving maximizes tokens served within the latency SLO; every lever — batching, chunking, disaggregation, speculation, routing — is a bet on where that goodput-optimal point sits for your model and traffic.