Efficient Request Queueing – Optimizing LLM Performance

Hugging Face Blog · 1 year ago

TNG Technology Consulting implemented fair scheduling at an API layer above inference engines like vLLM to prevent individual users from monopolizing GPU resources by filling the backend queue with many requests. The scheduler uses round-robin queuing per user and monitors backend queue length via Prometheus metrics, limiting new requests when queue depth exceeds a threshold such as three requests. This approach prevents latency spikes for new users while maintaining GPU efficiency through batch processing, with optional extensions based on token generation speed or request priority levels.