smevals - a small eval suite for evaluating models, prompts, and harnesses
Simon Willison Simon Willison
A researcher released smevals, a framework for evaluating AI models and prompts across different configurations by running tests and grading results. The tool uses YAML-based eval suites that can be tested against multiple models like GPT and Claude, with results viewable through a web interface or static HTML reports. This enables teams to systematically benchmark model capabilities and compare performance across different versions and configurations.
Why it matters
smevals - a small eval suite for evaluating models, prompts, and harnesses I've been working with Jesse Vincent's Prime Radiant applied AI research lab building out this evals framework to help answer questions about the capabilities of different models. The result is smevals, a new tool for running small eval suites across different model configurations and grading the results. This blog entry describes the tool in detail. Here's the 10 second version: Tell your coding agent to run uvx smevals docs to learn the tool (this outputs the README) Then tell it to build you an eval suite Once you've created an eval - which takes the form of a directory with some YAML files - you can run it against models like this: uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6 Runs are treated separately from grading operations - you can grade your runs (against your defined set of checks) using: uvx smevals grade path-to-eval/ Then you can run a localhost web server to explore the results: uvx
