Hyperscaler XPUs: TPU, Trainium/Inferentia, Maia, MTIA

The previous two chapters covered silicon you can buy: NVIDIA and AMD sell merchant accelerators to anyone with a purchase order. This chapter is about silicon you can only rent, or only build — the hyperscaler XPUs. Google's TPU, AWS's Trainium and Inferentia, Microsoft's Maia, and Meta's MTIA are not products on a price list. They are vertically-integrated cost structures: a chip co-designed with a compiler, wired into a proprietary fabric, deployed in racks the same company owns, serving models the same company (or its anchor tenant) runs. The 2026 reality is that roughly 28% of AI server shipments are now ASIC-based — the highest share since 2023 — and custom-ASIC unit growth is running near 45% year-over-year, nearly triple the rate of merchant GPUs (TrendForce / Tom's Hardware, May 2026). The merchant-GPU monopoly is not collapsing, but the inference half of the market is quietly defecting.

For most operators the XPU is not a thing you select — it is a thing you inherit when you pick a cloud. You do not buy a TPU; you rent a Cloud TPU slice and accept XLA. You do not buy a Trainium; you rent a Trn instance and accept the Neuron SDK. So the forks here are second-order: stable-workload-vs-portability, captive-vs-merchant, rent-the-anchor's-economics-vs-build-your-own. This chapter traces each and ends on the structural trend that ties them together — the model owner deciding that the cheapest way to serve its own tokens is to design the chip that serves them.

The motive is never raw FLOPS — merchant GPUs win on peak FLOPS and will keep winning, because NVIDIA spreads its R&D across the entire market. The motive is the intersection of three pressures that only a model-owner feels at full force. First, cost-per-token at volume. When you serve hundreds of billions of tokens a day against a model architecture you control and rarely change, a fixed-function accelerator tuned for exactly that shape beats a general-purpose GPU on tokens/$/W — and at hyperscale, single-digit-percent efficiency gains are nine-figure annual line items. Second, supply independence. NVIDIA allocation is the binding constraint of the era (see Chapter 7.6 on HBM and Chapter 2.3 on long-lead procurement); a hyperscaler with its own tape-out and its own TSMC wafer-and-CoWoS allocation is not standing in NVIDIA's queue. Third, the NVIDIA margin. NVIDIA's gross margin is the hyperscaler's cost; eliminating it on the highest-volume, most-predictable workloads is the single largest TCO lever a model-owner has.

The consequence — the reason most operators should not build — is that all three pressures require scale to pay off. Custom silicon carries a $100M-plus NRE per generation, a multi-year lead time, a hard minimum-volume threshold below which per-unit cost is worse than just renting GPUs, and a permanent software-engineering tax to keep the compiler competitive (see Chapter 7.5 for the ASIC break-even math). Only an entity that owns the model, owns the deployment, and ships the workload at hyperscale clears that bar. Everyone else rents the result.

The TPU is the only hyperscaler XPU with a decade of generations behind it, and it is the existence proof that the model-owner-builds-silicon thesis works at frontier scale — Gemini is trained and served on TPUs end to end. The 2026 flagship is Ironwood (TPU v7), made generally available at Google Cloud Next in 2026 and explicitly positioned as 'the first Google TPU for the age of inference.' Each chip delivers 4,614 FP8 TFLOPS and 192 GB of HBM3E at 7.37 TB/s — a 6x memory jump over Trillium (TPU v6), the largest single-generation memory expansion in TPU history — at roughly 2x the perf/watt of the prior generation (Google Cloud; SemiAnalysis, Nov 2025).

The architectural signature that separates TPU from every GPU fabric is the interconnect. TPUs are wired as a 3D torus over the Inter-Chip Interconnect (ICI, 9.6 Tb/s per chip on Ironwood), and the torus is stitched together by optical circuit switches (OCS) — Google's own MEMS-mirror optical switches that physically reconfigure which chips are neighbors. An Ironwood pod scales to 9,216 chips delivering 42.5 FP8 ExaFLOPS with ~1.77 PB of pooled HBM. The OCS decouples the logical topology a job sees from the physical wiring, which buys two things a switched fat-tree cannot: per-job topology-on-demand (a job requests a torus shape, the OCS provisions it), and fast routing-around-failures (a dead chip is optically bypassed rather than failing the slice). This is why Google can run ~9,200-chip coherent domains without the all-to-all switch silicon that dominates a comparable GPU cluster — the optics are the scale-up fabric. → Chapter 8.2 treats OCS-vs-switched scale-up in full.

The lock-in is correspondingly deep. There is no CUDA on a TPU; the path is JAX or TensorFlow lowered through XLA, the compiler that turns a high-level graph into TPU machine code. XLA is genuinely excellent for the dense and MoE transformer shapes Google cares about — but it is a compiler-first model, not a kernel-first one. You express computation and trust XLA to schedule it; you do not hand-write a PTX-equivalent kernel for a novel op the way a CUDA engineer does. For a stable architecture this is a feature (the compiler optimizes across the whole graph); for a research workload chasing a new operator every week it is friction. And TPUs are captive to Google Cloud — you cannot buy one, cannot colocate one, and your exit option is a full re-port to another stack. → Chapter 7.9 quantifies the XLA-vs-CUDA switching cost.

AWS runs a two-chip split: Inferentia for inference and Trainium for training, both programmed through the Neuron SDK, which sits under PyTorch and JAX. The 2026 story is Trainium3, announced at re:Invent 2025. Each Trainium3 chip delivers 2.52 FP8 PFLOPS with 144 GB of HBM3E at 4.9 TB/s — the first 3nm AI accelerator AWS has shipped. The system unit is the Trn3 UltraServer: up to 144 chips for 362 MXFP8 PFLOPS, 20.7 TB of HBM, and 706 TB/s of aggregate bandwidth, delivering ~4.4x the performance and 4x the perf/watt of the Trn2 UltraServer (AWS, Dec 2025). The scale-up fabric is NeuronLink-v4 with a NeuronSwitch-v1 all-to-all topology at 2 TB/s per chip — AWS taking the same vertical-scale-up page NVIDIA wrote, against an open-Ethernet scale-out (EFA).

What makes Trainium strategically different from TPU is the anchor-tenant model. Google builds TPUs primarily for itself; AWS builds Trainium substantially for Anthropic. Project Rainier — the Trainium2 cluster in St. Joseph County, Indiana, an $11B AWS site — went from announcement (re:Invent, Dec 2024) to a live cluster of ~500,000 Trainium2 chips in under twelve months, and by 2026 Anthropic and AWS report running over one million Trainium2 chips to train and serve Claude, with the partnership committed toward up to 5 GW of compute (AWS; Anthropic, 2026). External adopters such as Uber have cited cost savings on the order of 50% versus NVIDIA on suitable workloads. The anchor tenant de-risks the silicon: a guaranteed million-chip buyer amortizes the NRE before the first external customer ever rents a Trn instance. The consequence for everyone else: you are renting into economics that were sized for someone else's model, on a Neuron stack whose roadmap follows the anchor's needs, not yours.

The 'Access model' column governs strategy: every XPU here is captive, and two of them (MTIA, and effectively the first-party tiers of Maia) are not rentable at all. The choice of XPU is therefore downstream of a choice of cloud, and the choice of cloud is downstream of a choice of software stack you are willing to live inside. The perf/watt numbers matter, but the fork is which ecosystem you are entering.

The third cohort is the newest and the clearest signal of where the industry is heading. Microsoft Maia 200 deployed in early 2026: TSMC 3nm, 140B+ transistors, >10 PFLOPS FP4 / 5 PFLOPS FP8, 216 GB HBM3E at 7 TB/s in a 750 W envelope, which Microsoft claims delivers ~30% better performance-per-dollar than the best hardware in its existing fleet. Maia ships with a co-designed system: a wider rack and the Sidekick closed-loop liquid-cooling sidecar that lets Azure retrofit Maia into existing halls without re-plumbing the building (a deliberate density-ramp hedge — see Chapter 5.10 on liquid retrofits). In 2026 Maia 200 serves GPT-class models for OpenAI and powers Microsoft 365 Copilot. Meta MTIA took the opposite, most-aggressive roadmap stance: in early 2026 Meta disclosed four new generations (MTIA 300 through 500) for deployment through 2027, moving to 3nm with CoWoS packaging — silicon built for Meta's recommendation and GenAI inference and never sold to anyone.

The most consequential entrant is the one that is not a cloud at all. OpenAI, partnered with Broadcom in a ~$10B program, taped out its first custom inference ASIC ('Jalapeño') in roughly nine months, targeting deployment from late 2026 (VentureBeat / Tom's Hardware, 2026). This is the model-owner-builds-silicon trend in its purest form: the entity that owns the most-served model in the world deciding that the cheapest tokens are the ones running on a chip it designed. And it reveals the quiet kingmaker — Broadcom is the design-and-SerDes partner behind Google's TPU, Meta's MTIA, Microsoft's Maia, and OpenAI's Jalapeño (with Marvell playing a similar role elsewhere). The merchant-silicon disruption is not a swarm of independent chip startups; it is a handful of model owners renting Broadcom's and Marvell's packaging, SerDes, and physical-design expertise to convert their workload knowledge into captive silicon. → Chapter 7.5 on the merchant-silicon ASIC economics; Chapter 8.3 on the SerDes and switch-ASIC supply chain underneath all of it.

For an operator who is not a frontier model owner, the practical question collapses to three options, and the right one is a function of workload stability and your willingness to enter a single vendor's gravity well.

Stay on merchant GPUs (NVIDIA/AMD). The default, and correct for any workload that churns architecture, must run portably, or needs the deepest kernel ecosystem. You pay the NVIDIA margin and stand in the allocation queue, but you keep CUDA velocity and a multi-vendor exit. → Chapter 7.2, Chapter 7.3.

Rent an XPU through its captive cloud. Right when you have a stable high-volume inference workload and the XPU's tokens/$/W beats GPUs by enough to repay the port — a TPU slice for a JAX-native serving stack, a Trn/Inf instance for a PyTorch model that lowers cleanly through Neuron. The consequence is cloud lock-in: your serving stack now assumes that vendor's compiler, fabric, and roadmap, and your exit is a re-port. → Chapter 7.9, Chapter 7.11.

Build your own. Reserved for entities that own the model, ship at hyperscale, and can clear the $100M+ NRE, the multi-year lead time, and the permanent compiler-team headcount — almost always via a Broadcom/Marvell partnership rather than from scratch. The payoff is supply independence and a structurally lower cost-per-token; the risk is fixed-function rigidity against a model architecture that may move under you. → Chapter 7.5.