GPU Confidential Computing & Trusted Execution

Confidential computing answers a question that other controls cannot answer alone: how do you run on infrastructure you do not trust — and prove it? The tenant renting a GPU in a multi-tenant cloud, the sovereign government placing a frontier model in a hyperscaler region, the healthcare or finance customer whose data is the product — each either takes the operator's word that no privileged insider, no compromised hypervisor, and no co-resident tenant can read their weights and their prompts, or demands a cryptographic guarantee rooted in the silicon, enforced by hardware the operator itself cannot override, and attested end-to-end before a single byte of plaintext is released.

That guarantee is a Trusted Execution Environment (TEE) — simultaneously a strong cryptographic primitive and a leaky abstraction. Turn it on and you can exclude specified operator memory access within the supported CPU/GPU threat boundary, the hardest of the three states to protect. Turn it on without understanding what it does not cover and you have bought a false sense of security: plaintext metadata still leaks, side-channels still exist, the operator still controls scheduling and can deny service, and the entire chain rests on a firmware root of trust that someone has to attest. This is the canonical home for TEE and attestation in the guide; Chapter 10.3 (the sharing and resource-allocation model) and Chapter 11.6 (multi-tenant isolation) both point here.

Data exists in three states, and the security maturity of each is wildly different. At rest — weights on NVMe, checkpoints in object storage — is solved: AES-256 self-encrypting drives and KMS-wrapped keys are table stakes. In transit — across the fabric, between nodes — is solved: TLS, IPsec, and increasingly link-layer encryption (MACsec, NVLink encryption) close it. In use — the moment the weights are decrypted into GPU HBM and the prompts flow through the tensor cores — was, until recently, simply unprotected. A privileged host with root, a malicious hypervisor, a DMA-capable peripheral, or a cold-boot attacker could read plaintext model and data straight out of memory. Confidential computing exists to close that last gap, and only that gap. → the full at-rest / in-transit / in-use treatment for weights specifically lives in Chapter 11.8.

The mechanism is a hardware-enforced encrypted boundary. A CPU TEE (a confidential VM) encrypts and integrity-protects guest memory with a key held in the memory controller that no hypervisor or co-tenant can extract. A GPU TEE does the analogous thing for HBM and the on-die register/command interface. The two are chained: the confidential VM on the CPU and the confidential GPU context establish an encrypted, mutually-attested channel so that the bounce buffers used to move data across the (untrusted) PCIe bus are themselves encrypted. For an AI data center the load-bearing fact is that the model actually runs on the GPU — a CPU TEE alone protects nothing that matters. You need both, chained, attested, and that chaining is exactly where the cost and the caveats live.

The GPU TEE does not stand alone — it is anchored to a CPU confidential VM, and which one you get is set by your host platform, not chosen freely per workload. The two production options diverge in design philosophy, and that divergence cascades into attestation tooling and cloud availability.

AMD SEV-SNP (Secure Encrypted Virtualization — Secure Nested Paging) encrypts each VM's memory with a per-VM key managed by the on-die AMD Secure Processor, and adds Secure Nested Paging to defeat the hypervisor's ability to remap, replay, or alias guest pages — closing the integrity gap that earlier SEV generations left open. Its model is per-VM memory encryption with a relatively coarse, VM-granular boundary. Intel TDX (Trust Domain Extensions) builds a "trust domain" enforced by a dedicated, Intel-signed firmware module (the TDX Module) running in a new SEAM mode, mediating every transition between the untrusted hypervisor and the protected guest. TDX leans on a measured, attestable firmware mediator; SEV-SNP leans on the Secure Processor. Both produce a signed attestation report describing the launch measurement of the confidential VM. The practical consequence for an operator: your accelerator host CPU determines your TEE — an AMD EPYC host pairs with SEV-SNP, an Intel Xeon host with TDX — and the GPU confidential-computing stack, the attestation verifier, and the cloud regions where it is GA all follow from that one platform decision.

Making a GPU confidential is a different engineering problem than making a CPU confidential, because a GPU is a wide-open device by design: hundreds of memory-mapped registers, DMA engines that move data autonomously, and a command interface the driver pokes directly. The teardown of NVIDIA's implementation (independent analysis in arXiv 2507.02770, alongside NVIDIA's own WP-12554) shows three load-bearing mechanisms.

The Compute Protected Region (CPR) carves off the large majority of GPU memory — roughly 90% of HBM — into a hardware-access-controlled region that only the confidential GPU context can read. On Blackwell that region is also hardware-encrypted; on the studied Hopper configuration it is fenced rather than encrypted at runtime, and it is the transfers across the untrusted host boundary that are cryptographically protected. Weights and activations live here; the host is denied access. The BAR0 decoupler addresses the register-exposure problem: in normal mode only about 8% of the GPU's memory-mapped registers are hidden from the host, but in confidential-compute mode the decoupler hides roughly 99.78% of them, collapsing the management interface the host can touch down to a tiny, audited surface. Encrypted transfers handle the data path: because PCIe itself is untrusted, every payload crossing it moves through AES-GCM-encrypted bounce/staging buffers, with the per-channel keys derived from an SPDM-negotiated master secret (Gu et al.’s Appendix B enumerates 44 derived keys across RPC, DMA, fault and workload channels in the tested Hopper implementation). The result is a GPU whose memory and management plane the operator cannot read, even with full host root.

The most consequential line in that table is GPU scope. Hopper's supported multi-GPU mode passes all eight GPUs of the documented four-NVSwitch, air-cooled HGX system into one confidential VM, but GPU-to-GPU NVLink traffic remains unencrypted. Blackwell encrypts peer-to-peer NVLink traffic for up to eight GPUs in one CVM. That makes Blackwell the choice when the confidentiality boundary must include intra-node scale-up traffic, but the documented scope is still one eight-GPU CVM, not a rack-scale domain. Rack-scale trusted execution is a Vera Rubin capability, so topology plans must distinguish today's node boundary from the Rubin-era rack boundary.

The production existence proof arrived in June 2026: Apple extended Private Cloud Compute onto Google Cloud, running Apple Intelligence workloads on NVIDIA confidential computing (Blackwell-class GPUs) with Intel TDX CPUs and Google's Titan root of trust — software attestation rooted in at least two independent vendors' roots of trust, a cryptographically verifiable append-only ledger of every PCC machine in the fleet, and published binaries (Apple SEAR, 2026-06-08; summer preview ramp). Consumer-scale confidential inference on third-party GPUs is now shipping architecture, not research. An independent serving-stack measurement the same summer put B300 BF16 matmul under GPU-CC at ~0.2% overhead (one stack, one op class) — consistent with the vendor's "under ~3%" bound and with the residual tax living in host-device boundary crossings, not the tensor-core path.

Encryption protects its defined path; remote key release additionally needs evidence of the intended confidential recipient. A TEE that you cannot prove is genuine, running the firmware you expect, in the configuration you expect, is just a black box asserting it is trustworthy. Attestation is the cryptographic proof — and it is an operationally demanding part of the whole system, because it inserts a hard dependency into your boot path that did not exist before.

The flow, in NVIDIA's implementation, runs roughly: the GPU presents a device identity chain (the studied implementation reports a 5-certificate chain rooted in a vendor-provisioned device identity key) and a set of measurements — 64 records in Gu et al.’s tested Hopper implementation (3 July 2025, §8), capturing firmware versions, configuration, and security state. A verifier (NVIDIA's Remote Attestation Service, NRAS, or a self-hosted equivalent) checks the identity chain's signatures and compares each measurement against the Reference Integrity Manifest (RIM) — the vendor-published golden measurements for that exact firmware build. Only if the required evidence passes identity, authenticity, freshness, revocation and approved-state policy does the verifier issue an attestation token, which the relying party (the key broker) accepts as the precondition for releasing the decryption keys that let the workload start. The CPU TEE goes through the analogous SEV-SNP or TDX attestation against its own verifier. Bind both CPU/CVM and every assigned GPU to the tenant, workload image/configuration, fresh challenge and recipient key before releasing workload secrets; the record count is not a completeness score.

Confidential GPU topology must match the documented CVM boundary. Hopper's supported multi-GPU mode assigns all eight GPUs of the documented four-NVSwitch, air-cooled HGX system to one CVM but leaves GPU-to-GPU NVLink traffic unencrypted; Blackwell supports up to eight GPUs per CVM and encrypts their peer-to-peer NVLink traffic. The decision is therefore whether the workload's confidentiality boundary must include intra-node scale-up traffic, not whether Hopper is limited to one GPU.

TEE-I/O resolves both at once. By protecting peer-to-peer NVLink traffic across NVSwitch with hardware-accelerated link encryption, it lets up to eight Blackwell GPUs in one CVM communicate over encrypted NVLink: inter-GPU collectives stay encrypted but never leave the TEE, never hit the PCIe chokepoint, and run at near line-rate. This is what collapses the confidential-computing performance tax — up to ~25% on Hopper's worst-case adversarial patterns (multiples only on pathological pure-transfer microbenchmarks) — to under ~3% on Blackwell's large-tensor workloads, and it enables protected multi-GPU communication within the documented CVM boundary. The cross-reference here is structural: the same NVLink scale-up domain that Chapter 11.6 analyzes as an isolation boundary between tenants is what TEE-I/O turns into a confidentiality boundary against the operator. The two chapters describe the same fabric under two different threat models.

Hardware TEEs are necessary but not sufficient — they protect memory, not the software lifecycle around it. Two pieces of plumbing turn a confidential VM into a usable confidential workload, and each carries its own decision.

Confidential containers (the Kata/CoCo lineage and its cloud equivalents) run each pod or container inside its own confidential VM, so the orchestrator — Kubernetes, the node agent, the cloud control plane — is moved outside the trust boundary. This matters because in a normal Kubernetes node, the kubelet and container runtime can read any container's memory; confidential containers break that. The cost is a fatter per-pod footprint (a VM per workload, not a namespace) and an attestation step injected into pod startup. The decision is granularity: confidential VM per node is cheaper but coarser; confidential container per workload is the strong isolation posture but heavier.

The key broker is the linchpin and the most common place to get the architecture subtly wrong. The pattern is attestation-gated key release: encrypted weights and data sit at rest; the workload boots into its TEE; it presents its attestation token to a key broker service; the broker validates the token (the right firmware, the right measurements, the right identity) and only then releases the decryption keys, into the attested enclave, where the operator cannot see them. The weights never exist in plaintext outside a proven-good TEE. Get this right and you have a genuinely strong story; get it wrong — release keys before validating attestation, or run the broker somewhere the operator can read its memory — and you have moved the crown jewels' keys into the very environment you were trying to defend against. Attestation proves genuine hardware in a measured state; it does not vouch for the code that receives the plaintext. The release policy has to name the approved image, configuration, workload identity, recipient key, device set, and evidence freshness, because a vulnerable or malicious approved workload discloses data from inside a perfectly valid TEE — the vendor's threat model excludes compromised workload software, and so should yours. This attestation-gated release is the canonical mechanism that Chapter 11.8 relies on for in-use weight protection, and the key-management discipline (HSM/KMS hierarchy, rotation, revocation) lives there.

Confidential computing is a strong primitive with a precisely-bounded threat model, and the failure mode in practice is almost never the cryptography. It is a defender who assumed the boundary covered more than it does. The literature on where confidential VMs fall short (arXiv 2503.08256, a systematization of CVM trust relationships) is required reading before you stake a sovereignty claim on a TEE.

• Metadata leakage. RPC payloads are encrypted, but queue headers, command-ring structures, and physical-address tables on the untrusted side remain plaintext. An operator who cannot read your data may still infer access patterns, memory layout, and operation sequencing from the metadata. For some workloads that is harmless; for others, access-pattern leakage is itself a meaningful side channel.
• Uncore and microarchitectural side-channels. The TEE protects the compute and memory of the protected context, but shared uncore blocks — media engines (NVENC/NVDEC/NVJPEG), DRAM frequency scaling, contention on shared caches and interconnects — can leak information across the boundary. These are the same class of channels that Chapter 11.6 documents bypassing MIG and MPS isolation; a TEE does not make them disappear.
• The trusted-but-unverified scheduler. The hypervisor is removed from the confidentiality boundary but not from the availability and scheduling path. It still decides when your VM runs, can starve it, can mount denial-of-service, and in some designs the firmware mediator (the TDX Module, the Secure Processor) is itself a large trusted-computing-base component you are attesting but not auditing.
• The firmware / RIM supply chain. The entire guarantee bottoms out in golden measurements published by the vendor and a firmware root of trust on the device. If the RIM is wrong, if the firmware signing is compromised, or if the silicon root of trust is subverted, attestation passes for a compromised platform. This is precisely why Chapter 11.4 (hardware root of trust, Caliptra/DICE, OCP S.A.F.E.) is the foundation underneath this chapter — the TEE is only as trustworthy as the firmware integrity story beneath it.
• Physical and supply-chain attacks below the TEE's model. Interposer attacks, advanced fault injection, and supply-chain tampering before provisioning are outside the standard CC threat model. A nation-state adversary with physical access is not stopped by a TEE designed to defend against a malicious-but-remote operator.

Confidential computing is the right tool when you have a named, unavoidable, untrusted party in your data path and a workload whose plaintext exposure you cannot otherwise prevent. For those cases, compare the selected TEE with changing the custody or execution arrangement. Blackwell-class TEE-I/O changes the protected data path; configuration and workload still determine its cost. A 2026 paired-run B200 benchmark puts a correctly configured confidential serving stack at ~1–3%, a stock stack at 30–40%, and eight-GPU confidential training at ~10–13% — the cost living almost entirely in encrypted collectives and host-boundary crossings, not in GEMM or HBM. Those are one-host measurements, so publish the scope of whatever benchmark you cite and make a customer-workload acceptance test the acceptance criterion: platform, enabled I/O path, software version, and transfer pattern set the number, not the generation label. The cost includes both cycles and operations: you now run an attestation verifier and RIM pipeline as Tier-0 infrastructure, you architect a key broker that must never leak, and you carry a documented residual surface — metadata, side-channels, a trusted scheduler, and a firmware supply chain — that you must defend with the rest of this Part rather than assume the TEE covered.