Decode
The memory-bandwidth-bound phase that generates output tokens one at a time after prefill.
Current numbers
GTC 2026 (reported)Groq 3 LPU chip (~500 MB SRAM, ~150 TB/s) → LPX rack (256 LPUs, ~128 GB aggregate SRAM) fills the disaggregated decode/FFN slot (no HBM); Rubin GPUs retain prefill + attention. Rubin CPX reportedly pulled at GTC 2026 (no official NVIDIA cancellation)
xPyDruntime-reconfigurable disaggregation: x prefill workers feeding y decode workers, re-balanced live
10–100xper-sample CPU decode cost of image/video vs text — the reason the GPU:CPU ratio must be set per modality