Multi-Tenant & Workload Isolation Security

Multi-tenancy is the business model of the entire neocloud and most enterprise platform teams: buy one expensive accelerator, rent fractions of it to many workloads, and amortize its paid capacity across tenants whose measured workloads leave shareable headroom. The economic gain depends on useful output after contention, reserve and recovery, and the security consequences of that sharing are routinely underpriced. The instant two tenants share a physical GPU, a memory controller, an L2 cache, an NVLink domain, or merely a host kernel and a container runtime, you have created a shared dependency to qualify — and the only question left is whether it is a strong boundary, a soft convenience dressed up as a boundary, or an outright leak. The work here is telling those apart before a tenant's KV-cache shows up in a neighbor's framebuffer.

The route: the multi-tenancy spectrum from bare-metal to time-slicing, with the documented isolation failures at each rung; the question every platform team mishandles — is MIG / vGPU / time-slicing a security boundary? — answered from the hardware reality rather than the marketing; memory hygiene on reassignment as an explicit, owned policy instead of an assumption; and the isolation-versus-utilization economics that govern the whole thing, with confidential multi-tenancy addressing a selected host threat and retaining its own sharing and side-channel limits. The canonical TEE and attestation machinery lives in Chapter 11.5; here we decide when you need it.

Tenant isolation is a spectrum, and each step down it trades a stronger boundary for higher utilization. Read it top to bottom as decreasing isolation strength and increasing density: the moment you stop giving each tenant a whole physical GPU, you start sharing silicon that was never designed as a confidentiality boundary.

Bare-metal-per-tenant is the strongest posture: a tenant gets whole nodes, whole GPUs, and ideally a whole rack or NVLink domain, with no hypervisor and no co-tenant on the same silicon. The blast radius is the tenant's own. This is what serious training customers and sovereign deployments demand, and it is what the strongest neocloud SLAs implicitly promise. The cost is utilization: a tenant who under-uses the hardware strands it, and you cannot backfill without reintroducing a co-tenant.

VM-per-tenant (whole-GPU passthrough) keeps the GPU undivided but interposes a hypervisor, giving each tenant a strong CPU-side boundary (SR-IOV / PCIe passthrough) while the GPU itself is still owned end-to-end by one VM at a time. The boundary you now trust is the hypervisor and the IOMMU, not the GPU. This is the workhorse of enterprise multi-tenant clouds and a genuinely defensible boundary — the residual risk is hypervisor escapes and the host-side GPU management stack.

MIG (Multi-Instance GPU) partitions a single physical GPU into up to seven hardware-isolated instances with dedicated SM and memory-system slices. Copy and media-engine allocation is product- and profile-specific, and compute instances inside one GPU instance share their parent's memory and engines; select a +me profile where dedicated media engines are required. This is the only fractional-sharing mode with a hardware-enforced partition, which is why it is the one rung of fractional sharing that can credibly be called a security boundary. The cost is rigidity (fixed geometries, no oversubscription) and the residual uncore side-channels discussed below.

vGPU (mediated virtualization) time-multiplexes a GPU across guest VMs via a host-resident vGPU Manager that schedules and mediates access. The VM boundary is real, but the GPU is now shared through a software mediator with privileged host code — and that mediator has been the source of the most consequential 2025 multi-tenant CVEs. vGPU can be layered on MIG (MIG-backed vGPU) to combine hardware partitioning with the VM boundary, which is the strongest practical fractional mode.

Time-slicing is the bottom rung: processes share one GPU with neither memory nor fault isolation. It is a utilization tool for trusted workloads, not a tenant boundary.

MPS is a separate cooperative-workload service. Volta-and-later clients have isolated GPU address spaces, but scheduling hardware, memory bandwidth, caches, and capacity remain shared. Standard MPS provides no error isolation; R610 static SM partitioning contains some SM-triggered faults but is not a general fault-isolation guarantee. Neither mode is suitable as a hostile-tenant boundary.

The 2024–2025 disclosure record contains real, assigned CVEs at every layer of the stack, and reading them is the fastest way to calibrate which boundaries hold. The pattern is instructive: the catastrophic breaks are almost never in the GPU partitioning itself, but in the control plane around it.

The control-plane escape (the real catastrophe). NVIDIAScape — CVE-2025-23266, CVSS 9.0, disclosed by Wiz in July 2025 — is a three-line container escape in the NVIDIA Container Toolkit: a malicious container sets LD_PRELOAD against an OCI createContainer hook and gains root on the host, from which it can read, steal, or tamper with the models and data of every other tenant on the shared machine. It affected Container Toolkit up to 1.17.7 and GPU Operator up to 25.3.1. Multi-tenant security is a full-stack property, and the runtime is the weakest, most-exposed link.

The mediator leak (vGPU). NVIDIA's July 2025 vGPU bulletin disclosed cross-VM issues in the vGPU Manager: CVE-2025-23290 (cross-VM information disclosure — a guest reading global GPU metrics influenced by neighboring VMs, the first publicly acknowledged leakage of co-tenant activity through the mediator) and CVE-2025-23285 (a guest consuming global resources to deny service to neighbors). The same bulletin patched stack buffer overflows in the vGPU Manager (CVE-2025-23283/23284, CVSS 7.8) enabling guest-to-host code execution. The mediated-virtualization boundary is real but it is privileged software, and privileged software has bugs.

The memory-remanence leak (affected local-memory stacks). LeftoverLocals — CVE-2023-4969, Trail of Bits, 2024 — recovered another process's data from uncleared GPU local memory across process and container boundaries on affected Apple, Qualcomm, AMD, and some Imagination devices, enough to reconstruct an LLM's responses (≈181 MB recoverable per query against a 7B model on llama.cpp). Tested NVIDIA devices returned zeros, and the disclosure did not evaluate global HBM, MIG teardown, or vGPU release. The control belongs at the local/shared-memory layer: use patched drivers and kernels that clear scratchpad on affected stacks.

The uncore side-channels (even MIG). Spy in the GPU-box (arXiv 2203.15981) demonstrated cross-GPU L2/NVLink channels on an eight-P100 Pascal DGX-1; it did not test MIG. Veiled Pathways demonstrated uncore channels based on DRAM-frequency contention, media engines under MPS, and shared PCIe I/O between MIG instances. MIG's partition is genuinely hardware-enforced for memory and compute, but it is not a proof of side-channel resistance. For data where timing leakage is in-scope, MIG is necessary and not sufficient — and neither is a GPU TEE, whose deployment threat model excludes side channels. Pick the mitigation from the specific shared resource and leakage mechanism: stop sharing the contended engine, dedicate the allocation, or partition the path — and test the chosen control against the actual attack rather than assuming a TEE established timing noninterference.

The most routine multi-tenant control is also the most overlooked: verifying the teardown path when a resource changes hands. Every time a whole GPU, MIG instance, vGPU slice, or container allocation moves between tenants, require the named mode's vendor-defined clearing or cryptographic teardown and test it on the deployed stack before reassignment. The decision you own is who produces that evidence and whether reassignment fails closed when verification does not pass.

Two postures compete: rely on the documented driver/firmware teardown or gate reassignment on an operator-controlled verification test. The former minimizes handoff latency but inherits the vendor implementation; the latter adds measured scheduler delay and produces evidence for the tenant. For confidential workloads, verify that the platform's attested teardown destroys the protecting keys before treating residual ciphertext as inaccessible (Chapter 11.5).

When the data class genuinely cannot tolerate exposure to a co-tenant or to the operator — regulated health and financial data, sovereign workloads, frontier weights rented on infrastructure you do not own — the only sufficient answer is confidential computing: a per-tenant Trusted Execution Environment spanning the CPU (SEV-SNP / TDX) and the GPU (NVIDIA Hopper/Blackwell CC with an access-fenced HBM region (encrypted on Blackwell), the BAR0 decoupler, and encrypted transfers), gated by attestation so a tenant releases keys only to a measured, verified configuration. The full machinery — Compute Protected Region state, attestation via NRAS/RIM, TEE-I/O across NVLink, the residual attack surface — is the canonical subject of Chapter 11.5. The decision here is whether you need it, because it is not free.

Strong VM isolation + disciplined hygiene defends against an honest-but-curious neighbor and an accidental leak; it does not defend against a malicious co-tenant exploiting a side-channel, nor against a compromised or coerced operator, nor against an insider with host access. Confidential multi-tenancy moves the operator out of the trust boundary and makes the attestation the gate — at the cost of a measurable performance tax (encrypted PCIe/NVLink transfers and TEE overhead, larger for small-transfer chatty workloads than for large compute-bound ones), reduced GPU-sharing flexibility (CC modes constrain partitioning), and real operational complexity in key brokering and attestation-policy management. You pay goodput and ops to remove the operator from the trust model. If your tenants do not require that removal, you are buying nines of confidentiality the data class does not value — the security analog of over-provisioned redundancy.

Every rung you climb for stronger isolation costs utilization, and utilization is the entire economic premise of multi-tenancy. This is the central tension and it is quantifiable. A bare-metal tenant who drives a $30k+ accelerator at 40% leaves ~60% of it idle and unbackfillable. MIG recovers much of that by packing up to seven isolated tenants onto one die. Time-slicing recovers the most by oversubscribing — at the price of being no boundary at all. Confidential computing climbs back down the utilization ladder: it constrains partitioning, taxes transfers, and complicates scheduling, so the same hardware serves fewer effective tenant-hours.

The decision is therefore not "how secure can we be" but "what is the cheapest posture that is sufficient for this data class's adversary model" — which is why the table above is organized by adversary, not by feature. Over-isolating is a real cost: a commercial inference fleet serving public, non-sensitive content that runs everything under per-tenant confidential computing is burning goodput and capex to defend against a threat its data class does not face. Under-isolating is a breach: a regulated-data tenant time-sliced onto shared silicon with a stranger is one un-scrubbed allocation away from a disclosure. The ClusterMAX 2.0 rubric (SemiAnalysis) bakes this judgment into how the market grades neoclouds — tenant and fabric isolation are scored as first-class alongside goodput and health-checks, because buyers have learned to price the boundary, not just the FLOPs.

The pragmatic 2026 default for a serious multi-tenant operator is a tiered offering: bare-metal or VM-per-tenant for customers who pay for it and demand it; MIG-backed vGPU with verified hygiene as the standard fractional product; time-slicing reserved strictly for a single tenant's own co-operative workloads (never across trust boundaries); and confidential computing as a premium tier for regulated and sovereign demand. Selling time-slicing as a cross-tenant boundary, or selling "isolation" without naming which rung, is the misrepresentation that turns a CVE into a liability.