AMD Instinct & the Open Challenger

For most of the modern accelerator era the AMD question answered itself: the hardware was competitive on paper, the software was not, and a frontier operator could not afford to discover the difference on a production training run. That is no longer the safe default. As of mid-2026 AMD ships a part — the MI355X — that matches or beats NVIDIA's Blackwell generation on HBM capacity and FP4 throughput, has a hyperscaler anchor (the OpenAI 6 GW agreement) and a sovereign-cloud anchor (Oracle's 50,000-GPU MI450 commitment) underwriting its roadmap, and is bringing a 72-GPU open rack — Helios, built on Meta's OCP Open Rack Wide — to challenge the one place NVIDIA was uncontested: the rack-scale coherent domain. The silicon argument for a second source is largely won. The argument that remains is about realized performance, software maturity, and the timing of an open fabric standard — and that argument is where the money is actually made or lost.

This chapter walks the product line — MI300X, MI325X, MI355X, and the rack-scale MI400/MI450 family inside Helios — and the scale-up bet AMD is making with UALink-over-Ethernet. The practitioner's fork is where AMD wins today, and what the ROCm-maturity tax costs you when it does not. Paper FLOPS and HBM datasheets are necessary but not sufficient; the binding question is goodput — the fraction of that paper performance your software actually extracts — and whether the hardware discount survives contact with the porting bill.

AMD's strategy reads cleanly off the product cadence: lead with the axis NVIDIA was weakest on — memory capacity per GPU — because that is the axis that decides how many accelerators a large model needs to be sharded across, and therefore the axis that most directly drives inference cost-per-token. The MI300X (CDNA 3, launched December 2023) shipped with 192 GB of HBM3 at 5.3 TB/s and a 750 W TDP against the H100's 80 GB — a 2.4x capacity advantage that let a 70B-class model fit on a single GPU and a 405B-class model shard across one node instead of two. That is not a benchmark-chart win; it is a topology win, because fewer GPUs per model replica means less collective traffic, fewer failure domains, and a smaller KV-cache footprint per accelerator.

The MI325X (still CDNA 3, Q4 2024) widened the memory lead to 256 GB of HBM3E at 6 TB/s and 1,000 W — a mid-cycle capacity bump aimed squarely at NVIDIA's H200 (141 GB). The MI355X (CDNA 4, TSMC N3P, mid-2025) is the part that closed the compute gap: 288 GB of HBM3E at 8 TB/s, native FP6 and FP4 datatypes, roughly 20 PFLOPS of FP4 and ~10 PFLOPS of FP8 dense, at a 1,400 W liquid-cooled board power. That is HBM capacity ahead of a Blackwell GB200 die and FP4 throughput in the same class — for the first time, AMD is not conceding either of the two headline numbers. The precision story matters downstream: native FP4/FP6 is what makes the memory advantage pay off on modern inference, and it ties directly into the quantization tradeoffs in Chapter 7.10.

The fourth row is the one that changes the strategic picture. Single-GPU and single-node parts let AMD compete box-for-box — but NVIDIA's moat from the GB200 generation onward was never the box, it was the rack: a 72-GPU NVLink-coherent domain that software treats as one giant accelerator, which no competitor could match because no competitor had a scale-up fabric. The MI400/MI450 family inside Helios is AMD's answer. Each MI450X carries 432 GB of HBM4 at ~19.6 TB/s; a Helios rack fuses 72 of them into a single scale-up domain delivering ~260 TB/s of aggregate scale-up bandwidth, ~31 TB of HBM4, ~1.4 EFLOPS FP8 and ~2.9 EFLOPS FP4. On the headline rack numbers, that is NVL144-class. The catch is how the 72 GPUs are fused.

NVIDIA's NVLink is a proprietary, production-proven, purpose-built scale-up switch fabric. AMD cannot ship that, so it is betting on the open alternative: UALink, a consortium standard (AMD, Intel, Google, Meta, Microsoft, Broadcom, and others) finalized as UALink 1.0 — 200 GT/s per lane, load/store memory semantics, sub-1-microsecond round trips, up to 1,024 accelerators in a domain. On paper UALink is a genuine peer to NVLink. The problem is timing. True UALink switch silicon (from Astera Labs, Marvell, and others) is sampling and slips toward late 2026/2027 — which is after Helios needs to ship. So the first Helios racks run UALink's load/store protocol tunneled over Ethernet (UALink-over-Ethernet, UALoE) on Broadcom Tomahawk-class switches, not over native UALink switches.

The tradeoff is sharp. Tunneling over Ethernet gets AMD a 72-GPU coherent domain on schedule, riding the mature, multi-vendor Ethernet PHY and switch ecosystem instead of waiting on a brand-new switch ASIC. The downstream cost is that you are running a memory-semantic scale-up fabric over a substrate that was not designed for it, with a software and reliability stack that is younger than NVLink's by several production generations. NVLink's value was never the bandwidth number alone — it was that the failure modes, the firmware, the collective libraries, and the diagnostics were beaten into shape across two or three large deployments. An open fabric inherits none of that maturity for free. The full scale-up fabric comparison — NVLink vs UALink vs UALoE, domain sizing, and the copper/optics reach implications — lives in Chapter 8.2; here the point is narrower: committing a training campus's scale-up fabric to UALoE in 2026 is a bet that an open standard matures on your depreciation schedule.

The honest answer is workload-specific, and it splits along the training/inference line that governs everything else in Chapter 7.11. AMD's structural advantage is memory-capacity-bound inference. When the constraint is fitting a large model and its KV cache into HBM at a given concurrency, capacity-per-GPU is the lever, and AMD has led on it every generation. A 405B-class or 670B-class model that needs two H100 nodes can fit on one MI300X node; the larger HBM lets you run higher batch sizes before you spill, raising throughput-per-GPU. SemiAnalysis's independent inference benchmarking found exactly this texture: at small batch sizes and on the largest models (Llama 405B, DeepSeek V3/R1-class), MI300X delivers competitive or superior performance-per-dollar versus H100 — on the order of a ~20% cost-per-token advantage in those regimes — precisely because the memory advantage and AMD's hardware discount stack.

The mirror image is where AMD does not win: latency-sensitive, throughput-tuned inference against the H200, and tightly-coupled training. The same independent benchmarks show the H200 consistently delivering lower latency than MI300X across many configurations — not because of silicon, but because NVIDIA's TensorRT-LLM and CUDA collective stack extract more of the paper FLOPS than AMD's vLLM-on-ROCm path did at test time. That is the realized-MFU gap, and it is the entire reason the datasheet does not settle the question.

Here is the cost AMD's price discount is buying down. Paper FLOPS are an upper bound; goodput is what you pay for. Across the public record, MI300X has realized somewhere in the range of ~37-66% of H100/H200 effective performance on inference workloads despite carrying ~1.5x the paper FLOPS and more HBM bandwidth — the gap is software: less-optimized GEMM and attention kernels, a younger RCCL collective library, framework features that land on CUDA first, and a long tail of operators that work but are not yet tuned. ROCm 7.x has narrowed this materially (better vLLM/SGLang support, improved attention kernels, broader framework coverage), and AMD's day-zero model enablement has improved — but the gap is structural, not cosmetic: CUDA has a ~15-year head start, a deeper third-party library ecosystem, and the gravitational pull that every new model and kernel ships against it first. The full software-lock-in treatment, including how to quantify switching cost, is in Chapter 7.9.

The tax is paid in three currencies, and a TCO model that ignores any of them flatters AMD. First, engineering payroll: the FTEs who port, tune, and maintain the ROCm path — real money that only amortizes if spread across enough fleet. Second, realized utilization: every point of MFU you fail to extract inflates your effective cost-per-token, eating into the hardware discount. Third, schedule and risk: the new-model enablement lag and the thinner operational tooling translate into slower time-to-production and more debugging at scale. The discipline is to convert the discount and the tax into the same unit — cost-per-million-tokens at your realized utilization, not the datasheet's — and only then compare. → Chapter 7.11.

AMD's catch-up on FLOPS came the same way NVIDIA's did — by spending power. The MI300X's 750 W grew to the MI325X's 1,000 W and the MI355X's 1,400 W, and MI400/Helios is a liquid-cooled rack-scale part from the outset. The downstream consequence is identical to the NVIDIA density-ramp story in Part 5: at MI355X's 1,400 W the part is past the air-cooling cliff and direct-to-chip liquid is mandatory, and Helios — built on Meta's Open Rack Wide double-wide form factor with sidecar 400/800 VDC power — is a liquid, rack-scale integration unit, not a box you rack by hand. A buyer evaluating AMD on cost-per-token must therefore evaluate it on the same facility basis as NVIDIA: liquid cooling plant, high-density power chain, and reinforced floors. There is no air-cooled shortcut to the competitive AMD parts. The one genuinely differentiating facility choice is the open OCP rack lineage: Helios deliberately rides Meta's OCP contributions (Open Rack Wide, Mt. Diablo power), which is a hedge against single-vendor rack lock-in but does not change the underlying density/cooling physics. The cross-vendor cost-per-token model that decides all of this is built in Chapter 7.11; the consolidated per-generation roadmap is in Chapter 16.2.

The defensible position in mid-2026 is neither the bull's nor the bear's. AMD is a real second source — the first one the industry has had — and for memory-bound inference it is frequently the right economic choice on its own merits, before you even count the strategic value of supply leverage. For tightly-coupled frontier training it is still the higher-risk choice, gated less by silicon than by RCCL/collective maturity and by an open scale-up fabric (UALoE today, native UALink switches in 2027) that has not yet been proven at the scale and reliability NVLink has. The open question that decides AMD's trajectory is whether ROCm closes the realized-MFU gap and whether UALink hardware matures fast enough to make multi-vendor rack integration a genuine procurement hedge — and the anchor-tenant deals are the market's bet that it does. The practitioner's move is to capture the discount where the tax is already low, pursue the second source for leverage everywhere it breaks even, and keep the irreversible commitments (a training campus's scale-up fabric) on a shorter leash than the reversible ones (a portable inference stack) until the open fabric has shipped at scale.