VRAM · video RAM
The memory attached to a GPU, used loosely as a synonym for its HBM; it is the ceiling that weights and the KV-cache compete for during inference.
Current numbers
70–90%share of GPU VRAM the KV-cache consumes at 1M-token context
The memory attached to a GPU, used loosely as a synonym for its HBM; it is the ceiling that weights and the KV-cache compete for during inference.