Introducing AutoRound: Intel’s Advanced Quantization for LLMs and VLMs

Hugging Face Blog · 1 year ago

Intel released AutoRound, a post-training quantization method that compresses large language models and vision-language models by reducing bit precision while maintaining accuracy. At 2-bit precision, AutoRound achieves up to 2.1 times higher relative accuracy than competing methods, and quantizing a 72-billion-parameter model takes 37 minutes on an A100 GPU. The tool supports multiple export formats and device types, enabling efficient deployment of models on CPUs, Intel GPUs, and CUDA devices with minimal accuracy loss.