smevals - a small eval suite for evaluating models, prompts, and harnesses

Simon Willison has a new tool out called smevals, built while working alongside Jesse Vincent's Prime Radiant applied AI research lab. The goal is straightforward even if the problem it tackles isn't: figuring out, with some rigor, what different models are actually capable of. Rather than another giant benchmark suite, smevals is designed to be small and flexible, something you point at your own tasks rather than someone else's idea of what matters.

The workflow is deliberately simple. You tell a coding agent to run uvx smevals docs, which spits out the README and teaches the tool on the fly. From there you build an eval suite, which is really just a directory full of YAML files describing what you want tested. Running it against models is a one-liner too — something like uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6 — and the tool happily juggles multiple model configurations in a single pass.

Willison split running and grading into separate steps on purpose. Once you've got results, you grade them against whatever checks you defined, using uvx smevals grade path-to-eval/. After that you can spin up a local web server with uvx smevals serve to poke through the results interactively, or run smevals build to generate a static HTML report you can host wherever you like. He points to a haiku-writing eval he built as a working example of the whole loop in action.

What's notable here isn't the feature list so much as the backstory. Willison says he's been chasing a workable approach to evals for several years, and smevals is his third attempt at getting the idea right. This time, he says, it finally feels like it fits. He's already planning to expand it and turn it loose on his own projects, which suggests this isn't a one-off release so much as the start of a longer-running tool he intends to actually live with.