🚀 Accelerating LLM Inference with TGI on Intel Gaudi

Hugging Face Blog · 1 year ago

Text Generation Inference now includes native support for Intel Gaudi hardware accelerators in its main codebase, eliminating the need for a separate fork. The integration supports Gaudi1, Gaudi2, and Gaudi3 devices across AWS, Intel Tiber AI Cloud, and other platforms, with optimized implementations for 14 models including Llama 3.1, Mixtral, and Qwen2. Users can now deploy LLMs on Gaudi hardware with TGI's full feature set including dynamic batching, multi-card inference, and FP8 quantization without maintaining custom repositories.