TPOT · Time Per Output Token
The steady-state delay between successive generated tokens; the inter-token latency SLO governed by decode.
Current numbers
~40-50 msTPOT of ~20-25 tok/s — ~3-4x faster than typical human reading speed; common interactive decode target
up to ~77%goodput gain from hybrid aggregation/disaggregation over SOTA when both TTFT and TPOT bind