Bringing Nunchaku 4-bit Diffusion Inference to Diffusers

Modern text-to-image transformers are memory hogs. Loading one in BF16 can eat 20 to 30 GB of VRAM, which rules out most consumer GPUs before you've even generated a pixel. Diffusers already had answers for this — bitsandbytes, GGUF, torchao, Quanto — but all of them are weight-only schemes. They shrink the checkpoint on disk, then dequantize back to full precision to actually compute, which saves memory but does nothing for speed and sometimes adds a little latency.

SVDQuant, the method behind the Nunchaku engine, takes a genuinely different approach. It runs the core transformer layers with both 4-bit weights and 4-bit activations, so the math itself gets cheaper, not just the storage. The trick is handling outliers: SVDQuant shifts the ugly activation spikes into the weights, isolates the hardest part of each weight matrix into a small 16-bit low-rank branch, and quantizes what's left to 4 bits. Nunchaku's fused CUDA kernels then stitch the low-rank branch and the 4-bit path together so there's no extra memory-access penalty from carrying that 16-bit correction around.

Until now, using a Nunchaku checkpoint meant reaching for a separate inference library. Hugging Face's new Nunchaku Lite changes that by patching a stock Diffusers model's linear layers with runtime SVDQ or AWQ layers before the checkpoint loads, pulling the necessary kernels from the Hub via the kernels package — no local compilation. Two kernel families do the work: svdq_w4a4 handles the attention and MLP projections where most of the compute lives, in either INT4 or NVFP4, while awq_w4a16 covers precision-sensitive normalization and modulation layers with 4-bit weights and 16-bit activations. The catch is that the original Nunchaku engine gets extra speed from architecture-specific fused execution paths — fused QKV, fused GELU/MLP — that Lite doesn't replicate. So instead of matching that full engine's speedup, Lite delivers roughly a 30% speed gain while keeping the same memory reduction.

The numbers back it up. An ERNIE Image Turbo pipeline running Nunchaku NVFP4 alongside a bitsandbytes NF4 text encoder produced a 1024x1024 image in about 1.7 seconds on an RTX 5090, using roughly 12 GB of memory versus about 24 GB for the BF16 version. On an RTX PRO 6000, benchmarks show the BF16 baseline at 3.00 seconds full pipeline and 31.1 GB peak VRAM, dropping to 2.27 seconds and 20.6 GB with Nunchaku Lite NVFP4 — a 1.35x speedup. Adding torch.compile pushes that to 1.68 seconds, or 1.8x. Quantizing the text encoder too knocks peak VRAM down further, by about 22% in that test. NVFP4 needs a Blackwell-generation GPU; INT4 variants cover Turing, Ampere, and Ada cards, though Volta and Hopper aren't supported yet.

For people who want to quantize something themselves, the companion diffuse-compressor toolkit walks through calibrating, quantizing, packaging, and publishing a new model as an ordinary Diffusers repo. A FLUX.2 Klein 4B example finds 100 SVDQ targets, 3 AWQ targets, and 6 dense outer linears in that architecture. There's a real limitation, though: the generic path can't handle structural rewrites like fusing separate Q, K, and V projections into a single module, which is exactly what the original Nunchaku engine does for FLUX.1-dev to get its extra speed. Those cases still need a model-specific target config and adapter.