Optimum-NVIDIA Unlocking blazingly fast LLM inference in just 1 line of code

Hugging Face Blog · 2 years ago

Hugging Face released Optimum-NVIDIA, an inference library that accelerates large language model performance on NVIDIA GPUs through a modified single line of code. The library achieves up to 28 times faster throughput and generates 1,200 tokens per second using FP8 quantization on Ada Lovelace and Hopper architectures. Currently supporting LLaMA-based models, the library will expand to other architectures while adding techniques like In-Flight Batching and INT4 quantization.