HBM · High-Bandwidth Memory
Stacked DRAM mounted on-package with the accelerator; the bandwidth and capacity ceiling and the key supply bottleneck.
Vertically stacked DRAM bonded next to a processor for huge memory bandwidth; AI's binding supply constraint.
Also written as: High-Bandwidth Memory · HBM3 · HBM3E · HBM4
Current numbers
186 GBGB200 usable HBM per GPU
70 billion parameters × 2 B; 80 layers; 8 KV heads; head dimension 128; 2 B/KV value; 8,192 cached tokens; batch 8; workspace 20 GB; usable HBM 186 GB; sustained HBM 4.0 TB/s; step budget 50 ms.Fit one decode replica before buying more HBM — input ledger
About 180 GB resident; 40 ms memory lower bound. At most 9 sequences fit; 10 fail. Below about 3.2 TB/s the 50 ms memory budget fails.Fit one decode replica before buying more HBM — result and flip threshold