Introducing AutoRound: Intel’s Advanced Quantization for LLMs and VLMs
Hugging Face
Intel released AutoRound, a tool that shrinks LLMs and VLMs down to 2-8 bit weights without wrecking accuracy. It quantizes a 72B model in 37 minutes on one A100, which is stupidly fast.
Based on reporting by Hugging Face — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Quantization has become the unglamorous but essential plumbing of the AI world — nobody talks about it at parties, but without it you can't run a 70B model on anything short of a small data center. Intel's new tool, AutoRound, wants to make that plumbing better. It's a post-training quantization method that uses signed gradient descent to tune both weight rounding and clipping ranges at once, and the headline number is hard to ignore: at INT2 precision, AutoRound beats popular baseline methods by up to 2.1x in relative accuracy.
What makes this more than an academic exercise is the speed. Quantizing a 72B parameter model — Qwen2.5-72B, specifically — takes just 37 minutes on a single A100 GPU using AutoRound's "light" recipe. Compare that to AutoAWQ, which needed 230 minutes for the same job with a larger calibration set, or GPTQ via Transformers, which simply ran out of memory. AutoRound gets there with a calibration set of just 128 samples and 200 tuning steps, which is a fraction of what competing int2 methods typically demand.
The tool isn't limited to text models either. It covers more than ten vision-language models, including Mistral-Small-3.1 and Gemma3, and Intel has already pushed out ready-to-use quantized checkpoints through Hugging Face collections like OPEA and Kaitchup. Support spans CPUs, Intel GPUs, and CUDA, with export options for AutoRound's own format plus GPTQ, AWQ, and some GGUF variants — meaning existing GPTQ or AWQ checkpoints can be converted into AutoRound format rather than requantized from scratch.
Intel also built in three tuning presets — best, default, and light — trading time for accuracy depending on how much patience you have. On Qwen2.5-72B at 4-bit, the light recipe hits 0.7243 average accuracy across 13 benchmark tasks in 37 minutes, barely behind the 575-minute "best" run at 0.7242. That's a strange result on paper — the fast option essentially matching the slow one — but it says a lot about how much headroom there still is in quantization algorithms versus just throwing more compute at the problem.
My take — AI-written commentary, not fact-checked reporting
I run a lot of models on consumer hardware, so anything that makes INT2 usable without turning a model into mush gets my attention immediately. What I like here is that Intel isn't gatekeeping this behind their own silicon — it works on CUDA too, which is the only way an efficiency tool actually gets adopted instead of politely ignored. The real tell will be whether the community starts publishing AutoRound checkpoints as the default the way GPTQ and AWQ became default a couple years back.
Read more about this at: Hugging Face