Model & Weight Protection (At-Rest, In-Transit, In-Use)

Every other asset in an AI data center is replaceable. A GPU that fails is replaced from the spare pool; a transformer that trips is re-energized; a tenant's data is restored from backup. The weights are the exception. They are the distilled output of a training run that may have cost hundreds of millions of dollars and burned an interconnection slot you cannot get back — and unlike the hardware, they can be copied perfectly, instantly, and silently. The crown jewels are a few terabytes of floating-point that, once they leave, are gone with no way to claw them back and no way to know for certain they are gone.

The chapter follows the asset through its three states — at-rest (on disk, in checkpoints, in object storage), in-transit (across the fabric, between regions, to and from storage), and in-use (decrypted into HBM where a GPU can actually compute on it) — organized by RAND's Securing AI Model Weights and its five Weights Security Levels, the one published framework that ties controls to a named adversary tier rather than to a compliance checklist. Egress is treated as the choke point, attestation-gated key release as the in-use answer, and key management and PKI as a discipline rather than a feature; the close is the crypto-agility problem that long-lived weights force on you whether you want it or not. The data-governance and privacy regime — who is allowed to train on what — is a different problem; it lives in Chapter 10.10.

You cannot scope weight protection until you have named the adversary, and the industry's reference for naming adversaries is RAND's framework. It defines five Weights Security Levels (SL1–SL5), five operational-capacity tiers of attacker (OC1 amateurs through OC5 top-priority nation-state operations), and catalogs 38 distinct attack vectors. RAND defines each level by the attacker capability its safeguards are intended to thwart. SL2 addresses professional opportunistic efforts and explicitly excludes insiders; SL3 adds cybercrime syndicates and insider threats. Eight of the 38 vectors are likely infeasible for OC1–OC3 but likely feasible for OC4–OC5 — which is to say the threat-model jump from opportunistic attackers to top-tier operations is structural, not incremental.

That gap is why a cross-industry SL5 Task Force formed in March 2025, targeting nation-state-resistant frontier infrastructure by 2028/2029 and publishing Version 0.1 of an SL5 standard framed as a NIST SP 800-53 overlay (60 long-lead-time controls across 12 families, spanning network, physical, machine, personnel, and supply-chain streams). The reason the timeline is years out is the same reason siting and interconnection are: the controls with the longest lead time — facility construction, hardware procurement, organizational capability — must be planned before the steel is cut. SL5 is a building decision rather than a software patch, which is exactly why it belongs in this guide and not in a security-tooling appendix.

At-rest protection is the one part of the problem that is genuinely close to solved by mature practice, and it is therefore the part operators over-index on. Weights and checkpoints live encrypted on disk and in object storage under envelope encryption: a data-encryption key (DEK) encrypts the bytes, a key-encryption key (KEK) in an HSM or KMS wraps the DEK, and the HSM root never leaves tamper-resistant hardware. With exact illustrative 175 billion and 1 trillion parameter counts, the 14-byte recipe of BF16 weights, FP32 master weights and two FP32 Adam moments gives exactly 2.45 and 14 decimal TB; inspect the actual serializer before sizing encryption. At-rest encryption is bulk symmetric crypto (AES-256-GCM/XTS) where the performance cost is negligible against the storage bandwidth you already provisioned for checkpointing (Chapter 9.4).

The fork that matters at-rest is not whether to encrypt but where the key lives and who can release it. Disk-level transparent encryption with keys resident on the host protects against a stolen drive and nothing else — the moment an attacker has code execution on the host, the cleartext is theirs. The defensible posture binds DEK release to an attestation of the requesting node, so a checkpoint only decrypts on a machine whose firmware, kernel, and GPU state match a known-good measurement. That single design choice converts at-rest protection from 'encrypted until someone logs in' into 'encrypted until a verified machine asks' — and it is the same machinery that the in-use decision will reuse, which is why getting the key hierarchy right at-rest pays off twice. The deeper consequence: at-rest is the state where you have the most time and the fewest constraints, so it is where you should spend your strongest cryptography and your strictest access policy. It is also, bluntly, not where the weights get stolen.

An egress-rate limit or upload limit buys response time on the paths it actually controls. For exact assumed operands, a full 1,000 GB uncompressed artifact divided by aggregate allowed egress of 800 GB/day takes 1.25 days. This constructed ratio applies only to that complete payload and aggregate channel; a partial, compressed or adapted payload, parallel endpoints or a bypass can leak useful capability sooner. To make exfiltration take long enough to detect and interdict, the cap has to be aggressive — and that is exactly where it collides with the workload.

The collision is the inference-server paradox: a production inference fleet's entire job is to emit token output to users on the outside, continuously and at a volume that scales with traffic and revenue rather than with any security budget. A cap below the required aggregate inference output breaks the revenue service; a cap above that demand still needs controls on payload content, bypasses and parallel endpoints. So egress limiting works beautifully for a training cluster that has no reason to send weight-sized payloads outward, and works poorly for an inference cluster whose normal behavior is high-volume output. The 2026 frontier of this problem (active research, not settled practice) is inference-output verification — proving that what leaves is plausibly model output and not encoded model weights — and the looming counter-threat is aggressive weight compression: demonstrated work compresses weights to ~1.15 bit/parameter with minimal downstream loss — and exploratory results reach ~0.1 bit/parameter — in a theft context, shrinking the payload an attacker must exfiltrate and undercutting any fixed-rate cap. Egress limiting is necessary; on its own it is not sufficient.

In-transit protection on the wire is the least controversial of the three states: mutual TLS or IPsec for north-south and control-plane traffic, link-layer or fabric encryption for the high-bandwidth east-west path, and signed, hash-anchored manifests so a tampered checkpoint is detected before it is loaded. The decision here is where the encryption boundary sits relative to the fabric you are trying to keep fast. Encrypting the back-end collective fabric — the non-blocking InfiniBand or Spectrum-X plane that exists for all-reduce bandwidth — adds latency and, more importantly, may not be terminable at line rate without offload, so the common posture is to encrypt the storage and cross-region paths hard while relying on physical and segmentation controls (Chapter 11.7) for the in-rack scale-up fabric. The GPU-TEE answer (below) collapses part of this problem by extending the trust boundary across NVLink with TEE-I/O, so weights stay encrypted even on the inter-GPU link.

The subtler in-transit problem is provenance and integrity, not confidentiality. A weight blob that arrives intact but wrong — a poisoned checkpoint, a backdoored fine-tune, a substituted base model — is an integrity failure that encryption does nothing to catch. The control is a signed chain: every checkpoint and released weight artifact carries a cryptographic signature and a measurement that ties it back to the training run and the build pipeline, validated before load. This is the same measured-boot and golden-measurement machinery used for firmware integrity (Chapter 11.4), applied to the model artifact. Skip it and you have protected the weights from being read but not from being silently replaced — and a replaced model is, for many threat models, worse than a stolen one.

This is the hardest state and the one where the real decision lives, because to compute on weights a GPU must, at some instant, have them in usable form in HBM. The classic posture is plaintext-in-use behind a perimeter: decrypt the weights into HBM, trust that the host OS, hypervisor, BMC, network segmentation, and personnel controls keep everyone else out. It is fast, simple, and adds no TEE encryption path — and the cleartext sits in HBM for any principal whose privileges reach that memory. Against an OC4–OC5 adversary, or an insider with root, test which of those privileges the boundary actually excludes.

The alternative is attestation-gated key release into a GPU TEE: the weights stay encrypted until a confidential-computing enclave on the GPU proves, via a hardware-rooted attestation, that its firmware and configuration match a known-good measurement — only then does the KMS release the decryption key, and the weights decrypt inside the TEE-protected HBM region where even the host operator with root cannot read them. Release weight keys only after the required CPU/CVM and every assigned GPU evidence pass, bound to the authorized tenant, workload image/configuration, challenge and recipient key; Chapter 11.5 owns the boundary, attestation, performance, and residual-side-channel details, while this chapter owns the key-release decision.

Every control above — at-rest envelope encryption, in-transit mTLS, attestation-gated in-use release — collapses to the same root: a key hierarchy and the discipline that operates it. Get the keys wrong and the strongest cryptography in the world is a single stolen credential away from irrelevant. The hierarchy is conventional in shape and unforgiving in operation: a hardware root of trust (an HSM cluster, FIPS 140-3 Level 3 or equivalent) holds the top-level KEKs that never leave tamper-resistant silicon; those wrap per-domain and per-tenant keys in a KMS; those in turn wrap the DEKs that encrypt actual weight blobs. The decisions that distinguish a real program from a checkbox are about who can release, how fast you can revoke, and what a single compromise unlocks.

Authorization to release is where two-person control earns its keep: a weight-key release for the crown-jewel model should require an attestation and a policy that no single human or service credential can satisfy alone — the separation-of-duties and least-privilege discipline that Chapter 11.9 treats as the dominant unaddressed vector. Rotation and revocation at fleet scale is the operational nightmare nobody budgets for: re-wrapping DEKs across tens of thousands of accelerators, propagating a revocation before an attacker can use a leaked key, and doing it without stalling training goodput. A KMS that can issue a key in milliseconds but takes hours to revoke one across the fleet fails exactly when it matters — and be precise about what the operation buys: re-wrapping an unchanged DEK protects nothing from someone who already holds that DEK, and a revoked grant cannot recall a key already copied. Compromise response is re-encryption of the affected ciphertext under a fresh DEK, denial of future release, and termination or bounded leasing of the jobs already holding plaintext, with a stated recovery objective and a rehearsed checkpoint/restart path (Chapter 12.3). And blast radius: if one compromised KMS credential or one over-scoped IAM role can release every weight key, you have built a hierarchy with no interior walls — the same anti-pattern Chapter 11.7 warns about for the network, now applied to the keys. The control-plane secrets that operate this machinery (KMS credentials, attestation-service trust anchors, signing keys) are themselves crown-jewel-adjacent and belong under the same custody discipline as the weights they protect (Chapter 10.6, Chapter 11.7).

The last decision is the one easiest to defer and most expensive to get wrong, because its consequence lands years after the choice. Frontier weights are long-lived secrets — a model trained in 2026 may retain strategic and economic value for a decade. The asymmetric cryptography that wraps key material today (RSA, ECC) is the part of the stack a future cryptanalytically-relevant quantum computer threatens, and the threat does not wait for that computer to exist: harvest-now-decrypt-later means an adversary can exfiltrate today's RSA/ECC-protected key material and weight blobs now, store them, and decrypt them when the capability arrives. A one-week-lived session token may expire, but captured traffic from that session can contain long-lived secrets; token lifetime does not settle harvest-now-decrypt-later exposure. For a decade-lived weight, the encryption you choose in 2026 needs a planning horizon matched to the confidentiality lifetime, without assuming a known 2035 quantum capability.

The decision is therefore crypto-agility now: design the key hierarchy and the in-transit/at-rest envelope so the algorithms can be swapped without re-architecting, adopt NIST-standardized post-quantum KEMs and signatures (ML-KEM/ML-DSA) for the asymmetric layers protecting long-lived weight keys, and run hybrid classical-plus-PQC during the transition. The symmetric bulk encryption (AES-256) is already quantum-resistant enough at full key length, so the migration is concentrated in the key-wrapping and signing layers — which is exactly where attestation and PKI live. The consequence of deferring: you cannot retroactively protect weights that have already been harvested under classical crypto. This chapter is the canonical home for the PQC transition for long-lived weights.

Climbing the Weights Security Level ladder is not a matter of buying a product; each level demands controls with different lead times and irreversibility, and the most consequential ones are facility decisions that must be made before construction. In practice SL1–SL2 is operational hygiene you can add late, SL3 is a serious program you can retrofit with effort, and SL4–SL5 is a design basis you largely have to build in from the start.

• SL1 → SL2 (stop amateur and professional opportunistic efforts): at-rest encryption, RBAC and least privilege, MFA, audit logging, standard cloud hygiene. Cheap, late-bindable, reversible. The floor for any non-open-weight model.
• SL2 → SL3 (stop cybercrime and trained insiders): egress-rate limiting and upload caps, HSM-backed key custody, attestation-gated DEK release, two-person controls on weight-key release, behavioral monitoring, hardened offboarding. This is the first RAND level whose threat model includes insiders. Mostly retrofittable, but the egress and key-custody architecture is easier built-in.
• SL3 → SL4 (stop professional/state-adjacent operations): attestation-gated in-use protection as the default (GPU TEEs, confidential-by-default compute), hardware supply-chain assurance and firmware integrity below the root of trust, network microsegmentation with DPU enforcement, and personnel-security programs. This starts to dictate the hardware you buy and the topology you build — increasingly irreversible.
• SL4 → SL5 (stop top-tier nation-state operations): Version 0.1's 60 long-lead-time controls across 12 families — air-gapped or strongly-isolated compute for the highest-value weights, secure facilities, supply-chain provenance end-to-end, and organizational capability that takes years to build. This is the level the SL5 Task Force targets for 2028/2029 because, like interconnection, it is gated by lead time, not by will.

The recurring failure is the same one that governs density and power: designing the irreversible substrate to today's level and being unable to reach tomorrow's. If there is any chance a facility will host weights that attract OC4–OC5 attention, the confidential-compute capability, the HSM placement, the segmentation topology, and the secure-room provisions are headroom you reserve at scoping time — because, unlike the egress policy, you cannot retrofit them after the weights arrive.