Prefill and Decode for Concurrent Requests - Optimizing LLM Performance

Hugging Face Blog · 1 year ago · 21

Researchers at TNG analyzed how to handle multiple concurrent requests on language models by separating token generation into prefill (processing all prompt tokens in parallel) and decode (generating output tokens sequentially). Their self-hosted cluster of 24 H100 GPUs handles over 5,000 inferences per hour and generates more than ten million tokens daily across 50 applications. The choice between prefill-first scheduling (minimizes time to first token but interrupts ongoing generations) and chunked prefill (balances latency and throughput by processing prompts in segments) determines whether systems optimize for user experience or total token output.