Transformers now runs llama.cpp quants

Hugging Face is adding GGUF support to transformers, which means models packed for laptop-sized memory can be loaded with from_pretrained and used through the usual Python API. The pitch is simple: pick a GGUF file from the Hub, load it locally, and start generating without leaving transformers behind.

That matters because GGUF has become the default format for a lot of local AI work. It comes from the llama.cpp team, it’s widely used for inference, and checkpoints in that format have already been downloaded millions of times. The ecosystem around it is broad too: Hugging Face points to ready-to-use models from ggml-org, Unsloth, LM Studio Community, and bartowski, each offering different quantization choices for different machines.

Hugging Face is not pretending llama.cpp is obsolete. Quite the opposite. It says llama.cpp is still the recommended engine when efficient local inference is the main goal. What this update does is make GGUF models feel native inside transformers, so developers can inspect activations, tweak the forward pass, try custom decoding, evaluate quantized checkpoints, or even dequantize and fine-tune from a GGUF starting point.

The first target is Apple Silicon, starting with the Qwen3.5 architecture. Under the hood, transformers reuses ggml kernels through its kernels library and trims overhead in generate, which is how it gets closer to llama.cpp on local performance. The benchmark it shared on a MacBook Pro M2 Max with 32 GB of unified memory shows transformers close to llama.cpp across three GGUF checkpoints, though Hugging Face is careful to say the measurements are not identical because its test includes prefill while llama-bench reports decode-only throughput.

There’s also a broader play here. Once these kernels exist in PyTorch, they are not limited to GGUF files or even to text models. Hugging Face says the same building blocks could help with other architectures and modalities, including computer vision, audio, and multimodal models. That is the real story: not just local inference on a Mac, but a way to pull ggml’s speed tricks into the broader transformers world without forcing everyone into a separate runtime.