vLLM
A popular open-source inference engine known for PagedAttention and high-throughput continuous batching.
Current numbers
~$1.90/M tokself-hosted Llama-3.1-70B FP16 on 8× H100 at $2.40/GPU-hr; ~2,800 tok/s aggregate (vLLM, batch 256, 512-in/512-out)