HBM · High-Bandwidth Memory
Stacked DRAM mounted on-package with the accelerator; the bandwidth and capacity ceiling and the key supply bottleneck.
Vertically stacked DRAM bonded next to a processor for huge memory bandwidth; AI's binding supply constraint.
Also written as: High-Bandwidth Memory · HBM3 · HBM3E · HBM4
Current numbers
vast majority contractedMicron agreements cover the vast majority of its calendar-2027 HBM bit supply (September 30, 2026)
186 GBGB200 usable HBM per GPU
70 billion parameters × 2 B; 80 layers; 8 KV heads; head dimension 128; 2 B/KV value; 8,192 cached tokens; batch 8; workspace 20 GB; usable HBM 186 GB; sustained HBM 4.0 TB/s; step budget 50 ms.Fit one decode replica before buying more HBM — input ledger