AMD Instinct & the Open Challenger

For most of the modern accelerator era the AMD question answered itself: the hardware was competitive on paper, the software was not, and a frontier operator could not afford to discover the difference on a production training run. As of mid-2026 that calculus has shifted. AMD ships a part — the MI355X — that matches or beats NVIDIA's Blackwell generation on HBM capacity and FP4 throughput; it has a hyperscaler anchor (the OpenAI 6 GW agreement) and a sovereign-cloud anchor (Oracle's 50,000-GPU MI450 commitment) underwriting its roadmap; and it is bringing a 72-GPU open rack — Helios, built on Meta's OCP Open Rack Wide — into the one place NVIDIA was uncontested, the rack-scale coherent domain. The silicon argument for a second source is largely won. What remains is the argument about realized performance, software maturity, and the timing of an open fabric standard — and that is where the money is made or lost.

The product line runs MI300X, MI325X, MI355X, and the rack-scale MI400/MI450 family inside Helios, and the scale-up bet rides UALink-over-Ethernet. Two questions decide whether any of it is worth buying: where AMD wins today, and what the ROCm-maturity tax costs you where it does not. Paper FLOPS and HBM datasheets are necessary but not sufficient; goodput — the fraction of paper performance your software actually extracts — is the binding number, and it decides whether the hardware discount survives contact with the porting bill.

AMD's strategy reads cleanly off the product cadence: lead with the axis NVIDIA was weakest on — memory capacity per GPU — because that is the axis that decides how many accelerators a large model needs to be sharded across, and therefore the axis that most directly drives inference cost-per-token. The MI300X (CDNA 3, launched December 2023) shipped with 192 GB of HBM3 at 5.3 TB/s and a 750 W TDP against the H100's 80 GB — a 2.4x capacity advantage that let a 70B-class model fit on a single GPU and a 405B-class model shard across one node instead of two. That is not a benchmark-chart win; it is a topology win, because fewer GPUs per model replica means less collective traffic, fewer failure domains, and a smaller KV-cache footprint per accelerator.

The MI325X (still CDNA 3, Q4 2024) widened the memory lead to 256 GB of HBM3E at 6 TB/s and 1,000 W — a mid-cycle capacity bump aimed squarely at NVIDIA's H200 (141 GB). The MI355X (CDNA 4, TSMC N3P, mid-2025) is the part that closed the compute gap: 288 GB of HBM3E at 8 TB/s, native FP6 and FP4 datatypes, ~10.1 PFLOPS of FP4 dense (~20 with 2:4 structured sparsity) and ~5 PFLOPS of FP8 dense, at a 1,400 W liquid-cooled board power. That is HBM capacity ahead of a Blackwell GB200 die and FP4 throughput in the same class — for the first time, AMD is not conceding either of the two headline numbers. Native FP4/FP6 is what makes that memory advantage pay off on modern inference, and it feeds directly into the quantization tradeoffs in Chapter 7.10.

The fourth row is what changes the strategic picture. Single-GPU and single-node parts let AMD compete box-for-box, but from the GB200 generation onward NVIDIA's moat was the rack: a 72-GPU NVLink-coherent domain that software treats as one giant accelerator, which no competitor could match without a scale-up fabric of its own. The MI400/MI450 family inside Helios is AMD's answer. Each MI455X carries 432 GB of HBM4 at ~19.6 TB/s; a Helios rack fuses 72 of them into a single scale-up domain delivering ~260 TB/s of aggregate scale-up bandwidth, ~31 TB of HBM4, ~1.4 EFLOPS FP8 and ~2.9 EFLOPS FP4. On the headline rack numbers that is NVL72-class — the Rubin rack NVIDIA briefly marketed as NVL144. What remains contested is how those 72 GPUs are fused.

NVIDIA's NVLink is a proprietary, production-proven, purpose-built scale-up switch fabric. AMD cannot ship that, so it is betting on the open alternative: UALink, a consortium standard (AMD, Intel, Google, Meta, Microsoft, Broadcom, and others) finalized as UALink 1.0 — 200 GT/s per lane, load/store memory semantics, sub-1-microsecond round trips, up to 1,024 accelerators in a domain. On paper UALink is a genuine peer to NVLink; the constraint is timing. True UALink switch silicon (from Astera Labs, Marvell, and others) is sampling and slips toward late 2026/2027 — after Helios needs to ship. So the first Helios racks run UALink's load/store protocol tunneled over Ethernet (UALink-over-Ethernet, UALoE) on Broadcom Tomahawk-class switches, not over native UALink switches.

Tunneling over Ethernet buys AMD a 72-GPU coherent domain on schedule, riding the mature, multi-vendor Ethernet PHY and switch ecosystem instead of waiting on a brand-new switch ASIC. The cost is that you run a memory-semantic scale-up fabric over a substrate never designed for it, with a software and reliability stack younger than NVLink's by several production generations. What NVLink actually sells is not the bandwidth number — it is failure modes, firmware, collective libraries, and diagnostics beaten into shape across two or three large deployments, and an open fabric inherits none of that for free. The full scale-up comparison — NVLink vs UALink vs UALoE, domain sizing, and the copper/optics reach implications — lives in Chapter 8.2. The narrower point here: committing a training campus's scale-up fabric to UALoE in 2026 bets that an open standard matures on your depreciation schedule.

The answer is workload-specific, and it splits along the training/inference line that governs everything else in Chapter 7.11. AMD's structural advantage is memory-capacity-bound inference. When the constraint is fitting a large model and its KV cache into HBM at a given concurrency, capacity-per-GPU is the lever, and AMD has led on it every generation. A 405B-class or 670B-class model that needs two H100 nodes fits on one MI300X node; the larger HBM lets you run higher batch sizes before you spill, raising throughput-per-GPU. SemiAnalysis's independent inference benchmarking found exactly this texture: at small batch sizes and on the largest models (Llama 405B, DeepSeek V3/R1-class), MI300X delivers competitive or superior performance-per-dollar versus H100 — on the order of a ~20% cost-per-token advantage in those regimes — precisely because the memory advantage and AMD's hardware discount stack.

Where AMD does not win is the mirror image: latency-sensitive, throughput-tuned inference against the H200, and tightly-coupled training. The same independent benchmarks show the H200 consistently delivering lower latency than MI300X across many configurations — not from silicon, but because NVIDIA's TensorRT-LLM and CUDA collective stack extract more of the paper FLOPS than AMD's vLLM-on-ROCm path did at test time. This is the realized-MFU gap, and it is why the datasheet does not settle the question.

This is the cost AMD's price discount buys down: paper FLOPS are an upper bound, and goodput is what you actually pay for. Across the public record, MI300X has realized somewhere in the range of ~37-66% of H100/H200 effective performance on inference workloads despite carrying ~1.3x the paper FLOPS and more HBM bandwidth — the gap is software: less-optimized GEMM and attention kernels, a younger RCCL collective library, framework features that land on CUDA first, and a long tail of operators that work but are not yet tuned. ROCm 7.x has narrowed this materially (better vLLM/SGLang support, improved attention kernels, broader framework coverage), and AMD's day-zero model enablement has improved — but the gap is structural, not cosmetic: CUDA has a ~15-year head start, a deeper third-party library ecosystem, and the gravitational pull that every new model and kernel ships against it first. The full software-lock-in treatment, including how to quantify switching cost, is in Chapter 7.9.

The tax is paid in three currencies, and a TCO model that ignores any of them flatters AMD. First, engineering payroll: the FTEs who port, tune, and maintain the ROCm path — real money that only amortizes across a large enough fleet. Second, realized utilization: every point of MFU you fail to extract inflates your effective cost-per-token, eating into the hardware discount. Third, schedule and risk: the new-model enablement lag and thinner operational tooling translate into slower time-to-production and more debugging at scale. Convert the discount and the tax into the same unit — cost-per-million-tokens at your realized utilization, not the datasheet's — before you compare. → Chapter 7.11.

AMD bought its FLOPS the same way NVIDIA did — with power. The MI300X's 750 W grew to the MI325X's 1,000 W and the MI355X's 1,400 W, and MI400/Helios is a liquid-cooled rack-scale part from the outset. The facility consequence tracks the NVIDIA density-ramp story in Part 5: at 1,400 W the MI355X is past the air-cooling cliff, direct-to-chip liquid is mandatory, and Helios — Meta's Open Rack Wide double-wide form factor with sidecar 400/800 VDC power — arrives as a liquid, rack-scale integration unit, not a box you rack by hand. So a buyer evaluating AMD on cost-per-token has to evaluate it on the same facility basis as NVIDIA: liquid cooling plant, high-density power chain, reinforced floors. The competitive AMD parts offer no air-cooled shortcut. What does differ is the open OCP rack lineage: Helios rides Meta's OCP contributions (Open Rack Wide, Mt. Diablo power), a hedge against single-vendor rack lock-in that leaves the underlying density and cooling physics untouched. The cross-vendor cost-per-token model that decides all of this is built in Chapter 7.11; the consolidated per-generation roadmap is in Chapter 16.2.

The defensible read in mid-2026 lands between the bull and the bear. AMD is a real second source — the first the industry has had — and for memory-bound inference it is frequently the right economic choice on its own merits, before you count the strategic value of supply leverage. For tightly-coupled frontier training it is still the higher-risk choice, gated less by silicon than by RCCL/collective maturity and by an open scale-up fabric (UALoE today, native UALink switches in 2027) not yet proven at the scale and reliability NVLink has. What decides AMD's trajectory is whether ROCm closes the realized-MFU gap and whether UALink hardware matures fast enough to make multi-vendor rack integration a genuine procurement hedge — and the anchor-tenant deals are the market's bet that it does. So capture the discount where the tax is already low, pursue the second source for leverage wherever it breaks even, and keep the irreversible commitments (a training campus's scale-up fabric) on a shorter leash than the reversible ones (a portable inference stack) until the open fabric has shipped at scale.