HBM · High-Bandwidth Memory
Stacked DRAM mounted on-package with the accelerator; the bandwidth and capacity ceiling and the key supply bottleneck.
Current numbers
144 chipsper Trn3 UltraServer = 362 FP8 PFLOPS, 20.7 TB HBM, 4x perf/watt vs Trn2
GTC 2026 (reported)Groq 3 LPU chip (~500 MB SRAM, ~150 TB/s) → LPX rack (256 LPUs, ~128 GB aggregate SRAM) fills the disaggregated decode/FFN slot (no HBM); Rubin GPUs retain prefill + attention. Rubin CPX reportedly pulled at GTC 2026 (no official NVIDIA cancellation)
80 → 288 GB → ~1 TBHBM capacity per accelerator package: H100 80 GB; B200 180 GB; B300 288 GB (HBM3E); Rubin Ultra ~1 TB HBM4E on a four-chiplet package (roadmap)