How UK AISI and EvalEval Are Making Benchmark Results Reproducible
Hugging Face
UK AISI is publishing eval results through EvalEval’s system. That makes benchmark scores easier to verify, not just easier to quote.
Based on reporting by Hugging Face — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
The UK AI Security Institute is using EvalEval’s infrastructure to publish evaluation results in a more open, structured way. The point is simple: if a score matters, other people should be able to inspect how it was produced, not just see the number pasted into a blog post.
EvalEval has been pushing that idea with Every Eval Ever, a shared schema for evaluation reporting, and Evaluation Cards, which collects benchmark metadata, run data, and model metadata in one place. AISI’s participation gives the effort some real weight. This is not a theory paper about better documentation; it is a live release of results with setup information attached.
The shared data set covers five benchmarks from the main experiment in AISI’s paper, How Inference Compute Shapes Frontier LLM Evaluation: HealthBench, FrontierMath, Humanity’s Last Exam, SWE-Bench Pro, and Terminal-Bench 2.0. It includes six frontier models: Claude Opus 4, Claude Opus 4.5, Claude Opus 4.6, GPT-5, GPT-5.2, and GPT-5.4. There are also two related cyber evaluations, Cyber CTFs and The Last Ones, using a different, partially overlapping model set.
AISI’s paper looks at how benchmark performance shifts with inference-time compute and evaluation protocol. Humanity’s Last Exam is the clearest example in the release: performance changes depending on the setup, and the release shows the cumulative share of attempted tasks solved as token use rises. When models got correctness feedback from an oracle after each attempt, they kept solving more tasks as more tokens were spent.
That is the bigger story here. Open benchmark numbers are useful, but open benchmark numbers with enough context to replay the setup are much better. EvalEval wants more of the second kind, and AISI is now feeding that system with real results rather than slogans.
My take — AI-written commentary, not fact-checked reporting
This is the unglamorous stuff AI needs more of: boring structure, fewer magic numbers, and less scoreboard theatre. If a benchmark result can’t survive being put next to its setup, it was never much of a result. The industry still acts like transparency is optional decoration; it isn’t.
Read more about this at: Hugging Face