Network Segmentation, Microsegmentation & Zero Trust

Every preceding chapter in Part 11 hardened a component: the supply chain (11.3), the root of trust and firmware (11.4), the GPU TEE (11.5), the tenant boundary (11.6). This chapter hardens the connective tissue — the network that lets thousands of those components behave as one machine. And here the AI data center inverts the assumption every enterprise network security product was built on. The classic model is a fortified perimeter (north-south) protecting a soft interior; the interior is assumed friendly because crossing the perimeter was supposed to be hard. An AI cluster is almost entirely interior. More than three-quarters of data-center traffic is now east-west — workload-to-workload, GPU-to-GPU, host-to-storage, service-to-service — and on a training fabric that share approaches unity (Akamai / Gigamon, 2024-2026). The perimeter you spent your budget on guards a building whose interior is a single open floor.

So the segmentation decision is not "do we have a firewall." It is: when an attacker gets one foothold — a compromised inference container, a poisoned dependency, a credential lifted from a CI runner — how far can they go before they hit a wall? On a flat fabric, the answer is "everywhere the routing table reaches," and they get there fast: CrowdStrike measured average eCrime breakout time — initial access to first lateral movement — at 29 minutes in 2025, down from 48 the year before, with the fastest observed at 27 seconds and exfiltration beginning within four minutes in one case (CrowdStrike 2026 Global Threat Report). The discipline that answers the question is segmentation: macro- (coarse zones), micro- (per-workload policy), and the zero-trust posture that refuses to treat network location as a proxy for trust at all. This chapter walks those layers — where each wall costs goodput, where it costs nothing, and where leaving it out costs you the weights.

The single most common mistake is to reason about "the network" as one thing. An AI cluster has at least three physically and logically distinct fabrics, and each one wants a different segmentation strategy because each carries different traffic with different performance constraints. Conflate them and you either strangle the workload (inline enforcement on the wrong fabric) or leave the crown jewels exposed (no enforcement on the fabric that needs it).

The back-end / scale-out fabric (InfiniBand or RoCE/Spectrum-X Ethernet, → 8.4, 8.5) carries the collectives — all-reduce, all-gather, all-to-all — that dominate every training step. It is engineered for sub-2-microsecond latency and 1:1 non-blocking bisection bandwidth. Putting a stateful L7 firewall inline here is a category error: you would add latency to the one path whose latency directly sets goodput, and a single straggler link drags the whole synchronous job. The front-end / north-south fabric carries API traffic, ingress, internet egress, and tenant access — this is where the classic perimeter and inference-endpoint controls live, and where inline inspection is affordable because the traffic is comparatively sparse and latency-tolerant. The management / out-of-band (OOB) fabric (→ 8.7) reaches BMCs, PDUs, CDU controllers, switches, and the orchestration plane. It carries the least traffic and the most privilege — and it is the fabric whose compromise is catastrophic, because it is the path to firmware (11.4) and to physical-systems control (11.10).

Before microsegmentation there is macrosegmentation — the coarse partitioning of the cluster into a handful of trust zones with controlled choke points between them. This is the cheap, high-leverage layer, and it is mostly a build-time design decision: VRFs/VLANs on the front-end, PKeys (InfiniBand partition keys) or VXLAN/EVPN segments on the back-end, and a hard separation of the management network. The zones that matter for an AI cluster are recognizable: ingress/DMZ, inference-serving, training/compute, storage, management/OOB, and — where weights live — a high-value-asset enclave with the tightest egress of all.

The consequence of skipping macrosegmentation is the one that recurs across every breach postmortem: an attacker who compromises a low-value, internet-adjacent service (an inference container, a monitoring agent, a CI runner) finds nothing between them and the training fabric where the weights are loaded. The single most important coarse wall is between the workload that talks to the internet and the workload that holds the model. On the back-end fabric, InfiniBand partition keys are the native tool — but treat them as a segmentation control, not a strong security boundary: PKeys are enforced by the subnet manager and adapters, and like VLANs they are a configuration boundary an attacker who reaches the subnet manager or a misconfigured adapter can cross. That caveat — config boundaries are not cryptographic boundaries — is the bridge to why the industry moved toward DPU-enforced, identity-based policy.

Microsegmentation means the unit of policy is the individual workload, not the subnet — a flow from container A to container B is permitted only if an explicit identity-based rule says so, regardless of whether A and B share a VLAN. In a traditional enterprise this is done with host agents or hypervisor firewalls; both consume the very CPU cycles an AI host wants for data movement, and host agents live in the same trust domain as the workload they are supposed to police. The architectural answer that defines the 2026 AI data center is to move enforcement off the host and onto the DPU / SmartNIC at the network edge of every server (→ 8.3).

The DPU is the right enforcement point for three reasons. It sits on the wire between the host and the fabric, so every packet to and from the GPU server crosses it — there is no bypass. It runs its own isolated OS and Arm cores, so a fully compromised host (root on the x86 side, even compromised firmware) cannot disable or rewrite the policy executing on the DPU — the enforcement plane and the workload are in different trust domains. And it offloads the work to dedicated silicon, so stateful L4 filtering, microsegmentation, encryption, and flow telemetry run at line rate without stealing host cycles. NVIDIA's BlueField-4, launching with the Vera Rubin platform in 2026, lands this thesis at 800 Gb/s with 64 Arm cores and roughly 6x the compute of BlueField-3, and ships with an ecosystem (Check Point, Cisco, Palo Alto Networks, F5, Forescout, Armis, Trend Micro) building zero-trust, inline east-west enforcement on it (NVIDIA / HPCwire, Oct 2025). The trade here is sharp: choose host-agent microsegmentation and you pay in GPU-host CPU and accept a co-located enforcement plane; choose DPU enforcement and you pay in DPU cost and a new control plane to operate, but you get an enforcement point a compromised host cannot reach.

Zero Trust is the principle underneath all of the above, and it is widely mis-sold as a product you buy. The canonical definition is NIST SP 800-207 (Aug 2020): no implicit trust is granted to an asset based on its network location — "never trust, always verify." Every access request is authenticated, authorized, and (ideally) encrypted, on the working assumption that an attacker may already be inside the environment. For an AI cluster, that principle has three concrete consequences. First, workload identity replaces IP address as the basis of authorization: a flow is allowed because the calling workload proves who it is (mTLS, SPIFFE/SVID, signed tokens), not because it sits on the right subnet. Second, the management plane is never implicitly trusted from the data plane — reaching a BMC requires going through a policy-enforcement point and authenticating, even from inside the building. Third — and this is where zero trust and confidential computing converge — attestation becomes an access gate: a node is admitted to the trusted compute pool, and released a key, only after it proves its firmware and TEE state are measured and unmodified (the attestation flow is canonical in 11.5; key release in 11.8). Zero trust without attestation trusts a node's self-asserted identity; zero trust with attestation trusts a hardware-rooted proof of what the node actually is.

If you do one thing in this chapter, control egress. Every other control limits how far an attacker can move; egress control limits whether they can get the prize out. The crown jewel of a frontier cluster is the weights — hundreds of gigabytes to terabytes of model parameters whose theft is the entire point of most state-level intrusions (the asset framing is in 11.1; the protection regime in 11.8). The asymmetry of exfiltration is that the network was built for bandwidth: a cluster with terabit egress can ship a multi-hundred-GB weight set off-site in seconds if egress is default-allow. The single highest-leverage anti-exfiltration control is therefore a default-deny egress posture: the high-value enclave has no route to the internet except through an allow-listed, logged forward proxy, and large or anomalous outbound transfers from the weights enclave are blocked and alerted by policy rather than merely observed.

The trade is uncomfortable because default-deny egress breaks things: package installs, model-hub pulls, telemetry to SaaS, license checks. The cost of doing it right is an allow-list and a proxy you must maintain; the cost of not doing it is that your most expensive asset has an unmonitored exit. The mature pattern segments egress by zone — permissive for build/dev, default-deny with allow-list for production serving, and effectively air-gapped (data-diode-like, human-in-the-loop for any egress) for the weights enclave — and treats the egress proxy logs as a primary detection surface, not an afterthought. Egress control is where network segmentation, weight protection (11.8), and insider-threat defense (11.9) all meet at the same choke point.

The management and out-of-band fabric carries the least traffic and the most consequence. It reaches every BMC/IPMI controller, every switch management port, the orchestration plane (Slurm/Kubernetes control nodes), the telemetry pipeline (10.6), and — in a converged facility — the OT controllers for cooling and power (11.10). A foothold here is more than lateral movement to another workload; it is a path to firmware implantation (rewrite a BMC, persist below the OS, → 11.4) and, at the extreme, to physical-systems sabotage. The control is conceptually simple and operationally demanding: the management network is physically or strictly logically separated from every data-plane fabric, reachable only through hardened jump hosts under privileged-access management (PAM, → 11.9), with no path from a tenant workload to a BMC. The recurring failure is the convenience shortcut — a management VLAN that is routable from the production network "so the automation can reach the BMCs" — which collapses the most important air gap in the building for an operational nicety. The control-plane secrets that ride this fabric (BMC credentials, IPMI keys, orchestration tokens) are themselves a segmentation problem: they belong in an HSM/KMS-backed store (→ 11.8) reachable only from the management enclave, never embedded in images or reachable from the data plane.

The inference endpoint is the one surface deliberately exposed to the world, and it is where segmentation meets application security. The front-end fabric terminates ingress, so this is where the perimeter controls live: API gateway, authentication and rate-limiting per tenant, a WAF, and — increasingly — model-aware guardrails for prompt-injection and jailbreak attempts that the network layer cannot see. The segmentation discipline is to treat the serving tier as a semi-trusted DMZ: it talks to the internet, so it must be assumed reachable by attackers, and it must therefore be the most tightly segmented zone away from the weights. A compromised inference container should be able to load the model it serves and nothing else — no route to the training fabric, no route to other tenants' KV-caches, and default-deny egress so that a hijacked endpoint cannot become an exfiltration relay. This is the same blast-radius logic as the rest of the chapter, applied at the one place the blast is most likely to start.

The same mis-segmentations recur, each from treating the AI cluster like an enterprise LAN or from optimizing the wrong fabric:

• Perimeter firewall, flat interior. Spending the security budget on a hardened north-south edge while the 80%+ of traffic that is east-west crosses no wall at all. One interior foothold owns the fabric in minutes. The fix is interior segmentation, not a bigger perimeter.
• Inline L7 inspection on the collective fabric. Deep-packet-inspecting all-reduce traffic to "secure east-west" — and paying for it in tail latency and lost goodput on the one path that sets training economics. Segment the back-end with offloaded, lightweight controls; inspect at the egress and front-end.
• Default-allow egress from the weights enclave. Leaving the most valuable asset in the building with a terabit, unmonitored path to the internet. Default-deny with an allow-listed proxy is the single highest-leverage control and the one most often skipped because it is inconvenient.
• Routable management plane. Letting the BMC/OOB network be reachable from production "for the automation," collapsing the air gap that stands between a foothold and fleet-wide firmware compromise. The management plane is the attacker's actual objective; isolate it like it.
• Treating VLAN/PKey as a security boundary. Relying on configuration boundaries as if they were cryptographic isolation, when an attacker who reaches the subnet manager or a misconfigured adapter crosses them. Use them for segmentation; layer identity-based, DPU-enforced policy for actual containment.