Native-speed vLLM transformers modeling backend

Hugging Face · 3 months ago · 18

The transformers library's backend for vLLM now achieves inference speeds matching or exceeding vLLM's hand-written implementations through dynamic layer fusion and runtime optimization. Tests across three Qwen3 models—a 4B dense model, a 32B dense model with tensor parallelism, and a 235B mixture-of-experts model—demonstrated parity with native vLLM throughput on all benchmarks. Model authors can now deploy transformers implementations directly to vLLM with the `--model-impl transformers` flag without sacrificing performance or writing custom inference code.