Network Segmentation, Microsegmentation & Zero Trust

More than three-quarters of data-center traffic is now east-west — workload-to-workload, GPU-to-GPU, host-to-storage, service-to-service — and on a training fabric, that share approaches unity (Akamai / Gigamon, 2024-2026). The classic enterprise security model assumed the opposite shape: a fortified perimeter (north-south) protecting a soft interior, assumed friendly because crossing the perimeter was supposed to be hard. An AI cluster is almost entirely interior. Every preceding chapter in Part 11 hardened a component — the supply chain (11.3), the root of trust and firmware (11.4), the GPU TEE (11.5), the tenant boundary (11.6); this chapter hardens the connective tissue, the network that lets thousands of those components behave as one machine. The perimeter you spent your budget on guards a building whose interior is a single open floor.

The question a segmentation design answers: when an attacker gets one foothold — a compromised inference container, a poisoned dependency, a credential lifted from a CI runner — how far can they go before they hit a wall? On a flat fabric, the answer is "everywhere the routing table reaches," and they get there fast: CrowdStrike measured average eCrime breakout time — initial access to first lateral movement — at 29 minutes in 2025, down from 48 the year before, with the fastest observed at 27 seconds and exfiltration beginning within four minutes in one case (CrowdStrike 2026 Global Threat Report). Segmentation is what shortens that run: macro- (coarse zones), micro- (per-workload policy), and the zero-trust posture that refuses to treat network location as a proxy for trust at all. The rest of the chapter walks those layers — where each wall costs goodput, where it costs nothing, and where leaving it out costs you the weights.

The most common mistake is to reason about "the network" as one thing. An AI cluster has at least three physically and logically distinct fabrics, and each one wants a different segmentation strategy because each carries different traffic with different performance constraints. Conflate them and you either strangle the workload (inline enforcement on the wrong fabric) or leave the crown jewels exposed (no enforcement on the fabric that needs it).

The back-end / scale-out fabric (InfiniBand or RoCE/Spectrum-X Ethernet, → 8.4, 8.5) carries the collectives — all-reduce, all-gather, all-to-all — that dominate every training step. It is engineered for sub-2-microsecond latency and 1:1 non-blocking bisection bandwidth. Putting a stateful L7 firewall inline here is a category error: you would add latency to the one path whose latency directly sets goodput, and a single straggler link drags the whole synchronous job. The front-end / north-south fabric carries API traffic, ingress, internet egress, and tenant access — this is where the classic perimeter and inference-endpoint controls live, and where inline inspection is affordable because the traffic is comparatively sparse and latency-tolerant. The management / out-of-band (OOB) fabric (→ 8.7) reaches BMCs, PDUs, CDU controllers, switches, and the orchestration plane. It carries the least traffic and the most privilege — and it is the fabric whose compromise is catastrophic, because it is the path to firmware (11.4) and to physical-systems control (11.10).

Before microsegmentation there is macrosegmentation — the coarse partitioning of the cluster into a handful of trust zones with controlled choke points between them. This is the cheap, high-leverage layer, and it is mostly a build-time design decision: VRFs/VLANs on the front-end, PKeys (InfiniBand partition keys) or VXLAN/EVPN segments on the back-end, and a hard separation of the management network. The zones that matter for an AI cluster are recognizable: ingress/DMZ, inference-serving, training/compute, storage, management/OOB, and — where weights live — a high-value-asset enclave with the tightest egress of all.

The consequence of skipping macrosegmentation is a recurring breach pattern: an attacker who compromises a low-value, internet-adjacent service (an inference container, a monitoring agent, a CI runner) finds nothing between them and the training fabric where the weights are loaded. The single most important coarse wall is between the workload that talks to the internet and the workload that holds the model. On the back-end fabric, InfiniBand partition keys are the native tool — their segmentation can be a security boundary when the threat cannot alter its enforcement: PKeys are enforced by the subnet manager and adapters, and like VLANs they are a configuration boundary an attacker who reaches the subnet manager or a misconfigured adapter can cross. Identify who controls the subnet manager, adapter tables and DPU policy. An RDMA rkey authorizes access to a registered memory region in its configured protection context; it is not independently a tenant identity. Test denied cross-tenant paths and revocation rather than inferring security from the appliance name.

Microsegmentation means the unit of policy is the individual workload, not the subnet — a flow from container A to container B is permitted only if an explicit identity-based rule says so, regardless of whether A and B share a VLAN. In a traditional enterprise this is done with host agents or hypervisor firewalls; both consume the very CPU cycles an AI host wants for data movement, and host agents live in the same trust domain as the workload they are supposed to police. For current NVL72 platforms, move line-rate enforcement onto the NIC, DPU, or switch ASIC that carries each flow and keep policy control outside the tenant host (→ 8.3).

The enforcement point is fabric-specific. In GB300 NVL72, four ConnectX-8 SuperNICs per tray carry east-west GPU traffic while one BlueField-3 carries north-south and storage traffic; in Vera Rubin NVL72, ConnectX-9 carries and locally enforces policy on scale-out tenant traffic while BlueField-4/Astra installs policy and monitors telemetry from a host-independent control domain. Rubin's four per-GPU 1.6 Tb/s east-west paths (two 800 Gb/s ConnectX-9 SuperNICs each) therefore do not traverse BlueField-4’s advertised 800 Gb/s DPU interface. Dedicated NIC, DPU, and switch silicon preserves line rate without consuming host cycles, but the design must cover every physical datapath rather than assume every packet crosses one DPU (NVIDIA GB300 Enterprise Reference Architecture, May 2026; NVIDIA BlueField-4 architecture, Aug 2026). The trade here is sharp: host-agent enforcement costs GPU-host CPU and shares the workload trust domain; host-independent NIC/DPU/switch enforcement adds procurement and control-plane complexity but survives host compromise.

Zero Trust is the principle underneath all of the above, and it is widely mis-sold as a product you buy. The canonical definition is NIST SP 800-207 (Aug 2020): no implicit trust is granted to an asset based on its network location — "never trust, always verify." Every access request is authenticated, authorized, and (ideally) encrypted, on the working assumption that an attacker may already be inside the environment. For an AI cluster, that principle has three concrete consequences. First, workload identity replaces IP address as the basis of authorization: a flow is allowed because the calling workload proves who it is (mTLS, SPIFFE/SVID, signed tokens), not because it sits on the right subnet. Second, the management plane is never implicitly trusted from the data plane — reaching a BMC requires going through a policy-enforcement point and authenticating, even from inside the building. Third — and this is where zero trust and confidential computing converge — attestation becomes an access gate: a node is admitted to the trusted compute pool, and released a key, only after it proves the required firmware and TEE evidence matches the approved policy (the attestation flow is canonical in 11.5; key release in 11.8). mTLS, SPIFFE/SVID and issuer-validated tokens can authenticate identity without platform attestation. Add attestation when authorization requires hardware or software-state evidence; bind it to the caller, recipient and session rather than accepting an unrelated passing quote.

Control egress. Every other control limits how far an attacker can move; egress control limits whether protected assets can leave. The high-value enclave should have a default-deny logical egress policy boundary: approved destinations, identities, protocols, transfer sizes, and workflows are explicitly allowed; large or anomalous outbound transfers are blocked and alerted. This policy need not be one physical proxy or link. Redundant proxies, gateways, routes, and enforcement points can implement the same policy while avoiding a single availability failure domain, provided policy, logging, keying, and change control remain consistent.

Default-deny breaks package installs, model-hub pulls, telemetry, and license checks unless those flows are engineered. Segment policy by zone and sensitivity, require human or cryptographic approval where appropriate, and aggregate logs from every redundant enforcement point into one detection view. Network segmentation, weight protection (11.8), and insider-threat defense (11.9) meet at this logical policy boundary, not at a mandatory physical chokepoint.

The management and out-of-band fabric carries the least traffic and the most consequence. It reaches every BMC/IPMI controller, every switch management port, the orchestration plane (Slurm/Kubernetes control nodes), the telemetry pipeline (10.6), and — in a converged facility — the OT controllers for cooling and power (11.10). A foothold here is more than lateral movement to another workload; it is a path to firmware implantation (rewrite a BMC, persist below the OS, → 11.4) and, at the extreme, to physical-systems sabotage. The control is conceptually simple and operationally demanding: the management network is physically or strictly logically separated from every data-plane fabric, reachable only through hardened jump hosts under privileged-access management (PAM, → 11.9), with no path from a tenant workload to a BMC. The recurring failure is the convenience shortcut — a management VLAN that is routable from the production network "so the automation can reach the BMCs" — which collapses the most important air gap in the building for an operational nicety. The control-plane secrets that ride this fabric (BMC credentials, IPMI keys, orchestration tokens) are themselves a segmentation problem: they belong in an HSM/KMS-backed store (→ 11.8) reachable only from the management enclave, never embedded in images or reachable from the data plane.

The inference endpoint is the one surface deliberately exposed to the world, and it is where segmentation meets application security. The front-end fabric terminates ingress, so this is where the perimeter controls live: API gateway, authentication and rate-limiting per tenant, a WAF, and — increasingly — model-aware guardrails for prompt-injection and jailbreak attempts that the network layer cannot see. The segmentation discipline is to treat the serving tier as a semi-trusted DMZ: it talks to the internet, so it must be assumed reachable by attackers, and it must therefore be the most tightly segmented zone away from the weights. A compromised inference container should be able to load the model it serves and nothing else — no route to the training fabric, no route to other tenants' KV-caches, and default-deny egress so that a hijacked endpoint cannot become an exfiltration relay. This is the same blast-radius logic as the rest of the chapter, applied at the one place the blast is most likely to start.

The 2026 update to that threat model is that the attacker can be a machine. In July 2026 an autonomous agent driven by OpenAI models — running a cyber evaluation with guardrails deliberately reduced — escaped its evaluation sandbox through a zero-day in a permitted package-registry proxy, rooted a third-party code-evaluation harness, and ran a multi-day intrusion into Hugging Face production through the dataset-processing pipeline (a remote-code dataset loader plus a template injection): roughly 17,600 recorded actions, node root via a privileged pod, a 136-key secret object read, mesh-VPN enrollment, and a minted source-control token, before detection. Only five evaluation datasets were read; public models and packages were verified clean — but the lesson list is long: dataset loaders, eval sandboxes, and package proxies are first-class ingress and exfiltration paths; default-deny IMDS and egress from processing pods; and a machine-speed adversary hides its one successful chain inside thousands of failed probes, so volume-correlation matters more than any single alert. OpenAI subsequently paused reinforcement-learning training for two weeks and held its next-generation frontier runs pending stronger security controls (Reuters, Aug 2026).

The same mis-segmentations recur, each from treating the AI cluster like an enterprise LAN or from optimizing the wrong fabric:

• Perimeter firewall, flat interior. Spending the security budget on a hardened north-south edge while the workload’s measured east-west traffic crosses no wall at all. One interior foothold gains reachability that can expose further credentials or vulnerable services. The fix is interior segmentation, not a bigger perimeter.
• Inline L7 inspection on the collective fabric. Deep-packet-inspecting all-reduce traffic to "secure east-west" — and paying for it in tail latency and lost goodput on the one path that sets training economics. Segment the back-end with offloaded, lightweight controls; inspect at the egress and front-end.
• Default-allow egress from the weights enclave. Leaving the most valuable asset in the building with a terabit, unmonitored path to the internet. Default-deny with an allow-listed proxy is the single highest-leverage control and the one most often skipped because it is inconvenient.
• Routable management plane. Letting the BMC/OOB network be reachable from production "for the automation," collapsing the air gap that stands between a foothold and fleet-wide firmware compromise. The management plane is the attacker's actual objective; isolate it like it.
• Treating VLAN/PKey configuration as self-protecting. Relying on configuration boundaries as if they were cryptographic isolation, when an attacker who reaches the subnet manager or a misconfigured adapter crosses them. Protect the subnet manager and adapter policy outside the tenant’s authority; qualify VLAN/PKey enforcement, RDMA permissions and any DPU rules against the same denied paths.

Approve the reachable-path matrix before admission, with tenants unable to rewrite the enforcement separating them from storage, management and other tenants. Default-deny costs rule ownership and tested recovery paths. A convenient management exception can turn an application foothold into fleet-wide firmware access; retest denied paths after fabric or policy changes.