Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint

Simon Willison says the Bonsai 2 27B GGUF files can be tried from Prism’s Hugging Face repo, but there’s a catch: they need Prism’s own llama.cpp fork, not the stock build most people would reach for first. He even lays out the exact path, from grabbing the macOS runtime to downloading the model file and launching the server on port 8331.

The model file he points to is about 5.95 GB. For a 27B model, that’s the real headline here. Bonsai 2 27B is being framed as near-lossless compression in a footprint that is roughly nine times smaller, which is the sort of thing that makes local inference feel less like a hobbyist stunt and more like a practical option.

Willison says the bundled llama-server web UI is “very good,” and he also shows how to hit the local server through an OpenAI-style endpoint using the llm CLI. So this is not just a compressed model sitting on a shelf. It’s meant to be used.

Performance, though, looks a bit messy. On an M5 Pro, he saw around 20 tokens per second, then 44 after a server restart, and he says he’s not sure why the jump happened. On startup, the server reported that the tensor API wasn’t supported in that environment and disabled it. That’s a reminder that clever compression is one thing; a clean runtime is another.