BenchMIRT: What are LLM benchmarks actually measuring?

Hugging Face is highlighting a new way to audit LLM benchmarks by zooming all the way in to the level of individual prompts. The method is called BenchMIRT, and the basic idea is simple: a benchmark score can hide a lot of different signals, even when the test was designed to measure just one thing.

That matters because benchmarks are often treated as if they map neatly to a single capability. But the source gives a cleaner example of why that assumption breaks down. BBQ is meant to probe social bias, yet one of its questions about a grandson and grandfather booking an Uber also asks the model to keep track of who is who and reason from the evidence. So the score isn’t just about stereotypes. It’s also about whether the model can follow the setup.

BenchMIRT borrows from Item Response Theory, a psychometrics method for inferring abilities from patterns in test answers, then extends it with multidimensional IRT so it can separate more than one capability at once. The team trained it on results from 100 LLMs across 16 benchmarks and more than 34K questions, including six general-reasoning benchmarks and 10 from the Olmo 3 safety suite. They didn’t tell the system which benchmarks were supposed to measure what. It independently surfaced two main dimensions anyway: safety and general reasoning.

That split exposed some interesting mismatches. WMDP, which tests dangerous dual-use knowledge in areas like biology, chemistry, and cybersecurity, lined up more strongly with general reasoning than with safety. And because the benchmark rewards refusing or failing to give the dangerous information, stronger general reasoning was associated with lower WMDP scores. HarmBench also turned out to be mixed: its harmful prompts and contextual prompts both tracked more closely with safety, while its copyright questions were more tied to general reasoning.

The project also suggests a practical use beyond score-spotting. BenchMIRT can rank which questions are most informative and keep only a fraction of them. In the reported experiments, holding on to 10% of the questions usually preserved nearly the same picture of model strength, and 50% often matched the full benchmark even more closely. It also predicted whether a model would answer held-out questions correctly 79% of the time, versus 70% for a simpler baseline. That’s useful. But the team is also blunt about the trade-off: the same kind of insight that helps build tighter evaluations could help someone strip out the hard safety questions and make a benchmark easier to pass.