Introducing multi-backends (TRT-LLM, vLLM) support for Text Generation Inference

Hugging Face · 1 year ago · 21

Text Generation Inference now supports multiple inference backends including TensorRT-LLM and vLLM through a unified interface, allowing users to switch between different execution engines without changing their deployment code. The vLLM backend integration is scheduled for Q1 2025, with additional backends for llama.cpp, AWS Neuron, and Google TPU also in development. Users can now select the optimal backend for their specific hardware and performance requirements rather than being locked into a single inference solution.