Hyperscaler XPUs: TPU, Trainium/Inferentia, Maia, MTIA

The previous two chapters covered silicon you can buy: NVIDIA and AMD sell merchant accelerators to anyone with a purchase order. The hyperscaler XPUs you can only rent, or only build. Google's TPU, AWS's Trainium and Inferentia, Microsoft's Maia, and Meta's MTIA are not products on a price list; they are vertically-integrated cost structures — a chip co-designed with a compiler, wired into a proprietary fabric, deployed in racks the same company owns, serving models the same company (or its anchor tenant) runs. TrendForce's January 2026 forecast puts ASIC-based servers at roughly 28% of 2026 AI server shipments — the highest share since 2023 — with custom-ASIC unit growth running near 45% year-over-year, nearly triple the rate of merchant GPUs (TrendForce, January 2026; Tom's Hardware, May 2026). The merchant-GPU monopoly is not collapsing, but the inference half of the market is quietly defecting.

For most operators you do not select an XPU — you inherit one when you pick a cloud. You cannot buy a TPU; you rent a Cloud TPU slice and accept XLA. You cannot buy a Trainium; you rent a Trn instance and accept the Neuron SDK. The decisions here are therefore second-order: stable workload vs portability, captive vs merchant, renting the anchor's economics vs building your own. Underneath all of them runs one structural trend — the model owner concluding that the cheapest way to serve its own tokens is to design the chip that serves them.

Raw FLOPS alone do not decide: NVIDIA spreads R&D across merchant volume, while Google and AWS can optimize an XPU around their own workloads. What moves a model-owner is three pressures felt at full force. First, cost-per-token at volume. When you serve hundreds of billions of tokens a day against a model architecture you control and rarely change, a workload-specialized XPU that remains compiler-programmable through a stack such as XLA or Neuron can beat a general-purpose GPU on tokens/$/W when that workload lowers cleanly — and at hyperscale, single-digit-percent efficiency gains are nine-figure annual line items. Second, supply independence. NVIDIA allocation can be a binding constraint for a named delivery window (see Chapter 7.6 on HBM and Chapter 2.3 on long-lead procurement); a hyperscaler with its own tape-out and its own TSMC wafer-and-CoWoS allocation is not standing in NVIDIA's queue. Third, the NVIDIA margin. Avoiding NVIDIA's purchase price adds design, software, yield and sustaining costs; gross margin is not recoverable program savings.

All three pressures require scale to pay off, which is why most operators should not build. Custom silicon carries a program-specific NRE and development schedule, a hard minimum-volume threshold below which per-unit cost is worse than just renting GPUs, and a permanent software-engineering tax to keep the compiler competitive (see Chapter 7.5 for the ASIC break-even math). An entity with a durable workload, committed volume and a funded design/software organization can test that bar. An external renter instead qualifies the offered instance, region and reservation; internal-only silicon is not rentable capacity.

The TPU is the only hyperscaler XPU with a decade of generations behind it, and it is the existence proof that the model-owner-builds-silicon thesis works at frontier scale — Gemini is trained and served on TPUs end to end. The 2026 flagship is Ironwood (TPU v7), previewed in November 2025, made generally available March 31, 2026, and explicitly positioned as 'the first Google TPU for the age of inference.' Each chip delivers 4,614 FP8 TFLOPS and 192 GB of HBM3E at 7.37 TB/s — a 6x memory jump over Trillium (TPU v6), the largest single-generation memory expansion in TPU history — at roughly 2x the perf/watt of the prior generation (Google Cloud; SemiAnalysis, Nov 2025). Ironwood is what you can rent; the eighth generation is announced but not the same thing. On April 22, 2026 Google detailed TPU 8t — a pre-training part with a 9,600-chip 3D-torus superpod, SparseCore embedding offload, native FP4, and the two-layer Virgo scale-out fabric — and TPU 8i, a sampling-and-serving part with 3x the on-chip SRAM, a Collectives Acceleration Engine replacing Ironwood's SparseCores, and a high-radix Boardfly ICI topology that holds a chip-to-chip diameter of seven hops where a 1,024-chip torus needs sixteen. Read those as announced architecture, not schedulable capacity: the training/serving split changes the shape of the pod you will eventually buy, not the one you can buy now.

What separates TPU from every GPU fabric is the interconnect. TPUs are wired as a 3D torus over the Inter-Chip Interconnect (ICI, 9.6 Tb/s per chip on Ironwood), and the torus is stitched together by optical circuit switches (OCS) — Google's own MEMS-mirror optical switches that physically reconfigure which chips are neighbors. An Ironwood pod scales to 9,216 chips delivering 42.5 FP8 ExaFLOPS with ~1.77 PB of pooled HBM. The OCS decouples the logical topology a job sees from the physical wiring, which buys two things a switched fat-tree cannot: per-job topology-on-demand (a job requests a torus shape, the OCS provisions it), and fast routing-around-failures (a dead chip is optically bypassed rather than failing the slice). This is why Google can run ~9,200-chip coherent domains without the all-to-all switch silicon that dominates a comparable GPU cluster — the optics are the scale-up fabric. → Chapter 8.2 treats OCS-vs-switched scale-up in full.

The lock-in runs correspondingly deep. There is no CUDA on a TPU; the path is JAX or TensorFlow lowered through XLA, the compiler that turns a high-level graph into TPU machine code. XLA is genuinely excellent for the dense and MoE transformer shapes Google cares about — but it is a compiler-first model, not a kernel-first one. You express computation and trust XLA to schedule it, and where the compiler's choice is not good enough you drop into Pallas, JAX's kernel language for hand-written TPU kernels with explicit control over blocking and memory movement. What you do not get is a PTX-equivalent assembly layer or CUDA's two decades of third-party tuned kernels, so the honest comparison is supported ops, compiler constraints, profiling and performance portability — not custom kernels versus none. For a stable architecture this is a feature (the compiler optimizes across the whole graph); for a research workload chasing a new operator every week it is friction. And TPUs are captive to Google Cloud — you cannot buy one, cannot colocate one, and your exit option is a full re-port to another stack. → Chapter 7.9 quantifies the XLA-vs-CUDA switching cost.

AWS runs a two-chip split: Inferentia for inference and Trainium for training, both programmed through the Neuron SDK, which sits under PyTorch and JAX. The 2026 story is Trainium3, announced at re:Invent 2025. Each Trainium3 chip delivers 2.52 FP8 PFLOPS with 144 GB of HBM3E at 4.9 TB/s — the first 3nm AI accelerator AWS has shipped. The system unit is the Trn3 UltraServer: up to 144 chips for 362 FP8 PFLOPS, 20.7 TB of HBM, and 706 TB/s of aggregate bandwidth, delivering ~4.4x the performance and 4x the perf/watt of the Trn2 UltraServer (AWS, Dec 2025). The scale-up fabric is NeuronLink-v4 with a NeuronSwitch-v1 all-to-all topology at 2 TB/s per chip — AWS taking a vertical scale-up like NVIDIA's, against an open-Ethernet scale-out (EFA).

What makes Trainium strategically different from TPU is the anchor-tenant model. Google builds TPUs primarily for itself; AWS builds Trainium substantially for Anthropic. Project Rainier — the Trainium2 cluster in St. Joseph County, Indiana, an $11B AWS site — went from announcement (re:Invent, Dec 2024) to a live cluster of ~500,000 Trainium2 chips in under twelve months, and by 2026 Anthropic and AWS report running over one million Trainium2 chips to train and serve Claude, with the partnership committed toward up to 5 GW of compute (AWS; Anthropic, 2026). External adopters such as Uber have cited cost savings on the order of 50% versus NVIDIA on suitable workloads. The anchor tenant de-risks the silicon: a guaranteed million-chip buyer amortizes the NRE before the first external customer ever rents a Trn instance. By Q2 2026 the chip business had become a headline in its own right — ~$25B annualized revenue across Trainium, Graviton, and Nitro (blended, not Trainium-only), ~$225B of multi-year commitments, and Trainium4 pointed at 2027 (Amazon, Aug 2026). The anchor is no longer exclusive, though: Anthropic signed for up to 2 GW of AMD MI450/Helios in July 2026 (Chapter 7.3), so the Rainier tenant now dual-sources. The consequence for everyone else: you are renting into economics that were sized for someone else's model, on a Neuron stack whose roadmap follows the anchor's needs, not yours.

The 'Access model' column is what governs strategy: every XPU here is captive, and two of them (MTIA, and effectively the first-party tiers of Maia) are not rentable at all. Choosing an XPU is therefore downstream of choosing a cloud, and choosing a cloud is downstream of choosing a software stack you are willing to live inside. The perf/watt numbers matter, but what you are really picking is an ecosystem.

The third cohort is the newest, and the clearest signal of where the industry is heading. Microsoft Maia 200 deployed in early 2026: TSMC 3nm, 140B+ transistors, >10 PFLOPS FP4 / 5 PFLOPS FP8, 216 GB HBM3E at 7 TB/s in a 750 W envelope, which Microsoft claims delivers ~30% better performance-per-dollar than the best hardware in its existing fleet. Maia ships with a co-designed system: a wider rack and the Sidekick closed-loop liquid-cooling sidecar that lets Azure retrofit Maia into existing halls without re-plumbing the building (a deliberate density-ramp hedge — see Chapter 5.10 on liquid retrofits). In 2026 Maia 200 serves GPT-class models for OpenAI and powers Microsoft 365 Copilot. Meta MTIA took the opposite, most-aggressive roadmap stance: in early 2026 Meta disclosed four new generations (MTIA 300 through 500) for deployment through 2027, moving to 3nm with CoWoS packaging — silicon built for Meta's recommendation and GenAI inference and never sold to anyone.

The most consequential entrant is not a cloud at all. OpenAI, partnered with Broadcom in a ~$10B program, taped out its first custom inference ASIC ('Jalapeño') in roughly nine months, targeting deployment from late 2026 (VentureBeat / Tom's Hardware, 2026). This is the model-owner-builds-silicon trend in its purest form: the entity that owns the most-served model in the world deciding that the cheapest tokens are the ones running on a chip it designed. And it reveals the quiet kingmaker — Broadcom is the design-and-SerDes partner behind OpenAI's Jalapeño (with Marvell playing a similar role elsewhere). The merchant-silicon disruption is not a swarm of independent chip startups; it is a handful of model owners renting Broadcom's and Marvell's packaging, SerDes, and physical-design expertise to convert their workload knowledge into captive silicon. → Chapter 7.5 on the merchant-silicon ASIC economics; Chapter 8.3 on the SerDes and switch-ASIC supply chain underneath all of it.

For an operator who is not a frontier model owner, the practical question collapses to three options, and the right one is a function of workload stability and your willingness to enter a single vendor's gravity well.

Stay on merchant GPUs (NVIDIA/AMD). A candidate for workloads that churn architecture, need supported portable paths, or depend on its kernel ecosystem. You pay the NVIDIA margin and stand in the allocation queue; in exchange you keep CUDA velocity, and a multi-vendor exit still requires a tested ROCm or other backend. → Chapter 7.2, Chapter 7.3.

Rent an XPU through its captive cloud. The right call when you have a stable high-volume inference workload and the XPU's tokens/$/W beats GPUs by enough to repay the port — a TPU slice for a JAX-native serving stack, a Trn/Inf instance for a PyTorch model that lowers cleanly through Neuron. The consequence is cloud lock-in: your serving stack now assumes that vendor's compiler, fabric, and roadmap, and your exit is a re-port. → Chapter 7.9, Chapter 7.11.

Build your own. Reserved for entities that own the model, ship at hyperscale, and can clear the program-specific NRE and development schedule, and the permanent compiler-team headcount — almost always via a Broadcom/Marvell partnership rather than from scratch. The potential payoff is a separately contracted allocation and a lower cost-per-token after program costs; the risk is a workload-specialized design's compiler lock-in against a model architecture that may move under you. → Chapter 7.5.