BenchMIRT: What are LLM benchmarks actually measuring?

AI2 has a new tool for poking inside LLM benchmarks, and it’s called BenchMIRT. Instead of treating a benchmark score like a clean verdict, it looks at the individual prompts underneath it and asks what ability is actually driving the result.

That matters because benchmarks are rarely as single-minded as their labels suggest. BBQ is meant to probe social stereotypes, yet one of its questions about an Uber ride for a grandfather and grandson also asks the model to keep track of who is who and reason from the evidence. WildJailbreak mixes harmful jailbreak prompts with benign ones meant to catch models that refuse harmless requests too often. Put those together and one score can blur two very different things.

BenchMIRT uses ideas from Item Response Theory, a psychometrics technique built around a simple point: questions are not equally informative. Some are harder. Some separate strong models from weak ones better. AI2 extends that with multidimensional IRT, so the system can tease apart more than one underlying capability at once.

The team trained it on results from 100 LLMs, 16 benchmarks, and more than 34,000 questions. They did not tell it which benchmarks were supposed to measure safety or reasoning. It still recovered those two dimensions on its own, and when they reran the analysis, the same pair kept showing up.

That is where the paper gets interesting. BenchMIRT largely agreed with the usual picture for many benchmarks: reasoning tasks tracked reasoning, and jailbreak-style safety tasks tracked safety. But it also showed that BBQ, even though it is grouped with safety, lined up much more strongly with general reasoning. WMDP, which covers dangerous dual-use knowledge in biology, chemistry, and cybersecurity, also leaned more toward reasoning than safety. HarmBench was mixed too, with its standard and contextual prompts mapping more closely to safety, while its copyright subset did not.

The method can also trim fat. AI2 says keeping just 10% of questions often preserved the same model ordering as the full benchmark, and 50% sometimes matched even more closely. It also predicted held-out answers correctly 79% of the time, versus 70% for a simpler benchmark-average approach.

There are caveats. The models used were all released by March 2025, so this is not a read on newer systems. And the dimensions BenchMIRT finds depend on the benchmark set it is given. But the broader point is hard to miss: a benchmark score can hide a lot of unrelated machinery.

What BenchMIRT finds each benchmark measures

Each benchmark's correlation with the two abilities it identified, safety and general reasoning, across 100 open-weight LLMs.

BenchmarkGeneral reasoningSafetyAudit verdict Used to measure general reasoning MMLU-ProGeneral reasoning0.97*Safety-0.21Aligned with stated benchmark goal BBHGeneral reasoning0.94*Safety-0.20Aligned with stated benchmark goal GPQAGeneral reasoning0.81*Safety-0.12Aligned with stated benchmark goal IFEvalGeneral reasoning0.72*Safety-0.34*Aligned with stated benchmark goal MATHGeneral reasoning0.70*Safety-0.41*Aligned with stated benchmark goal MuSRGeneral reasoning0.67*Safety0.00Aligned with stated benchmark goal Used to measure safety WildJailbreakGeneral reasoning0.14Safety-0.90*Aligned with stated benchmark goal—though its small benign set (250 of 2,250 items) leans general reasoning JailbreakTriggerGeneral reasoning0.27*Safety-0.91*Aligned with stated benchmark goal Do-Anything-NowGeneral reasoning0.20Safety-0.88*Aligned with stated benchmark goal HarmBenchGeneral reasoning0.32*Safety-0.90*Aligned with stated benchmark goal—except its copyright subset, which leans general reasoning StrongRejectGeneral reasoning-0.16Safety-0.84*Aligned with stated benchmark goal WildGuardTestGeneral reasoning0.40*Safety-0.87*Aligned with stated benchmark goal XSTestGeneral reasoning0.46*Safety-0.53*Split between the two abilities, consistent with its even split of overrefusal and safety items ToxiGenGeneral reasoning0.40*Safety-0.32*Weak on both—a fairly saturated benchmark, with a 92% average score across models BBQGeneral reasoning0.85*Safety-0.06Tracks general reasoning despite being a safety benchmark—a low BBQ score may say more about a model's reasoning than its safety WMDPGeneral reasoning-0.89*Safety0.21Scores a model's lack of dangerous dual-use knowledge—so it runs opposite to general reasoning, and shows no significant correlation with safety

Bar size and direction show the Pearson correlation, across 100 open-weight LLMs, between BenchMIRT ability scores and benchmark scores on a -1 to 1 scale—pink for general reasoning and teal for safety; bars extending left of center are negative. Bold with underline marks each row’s stronger correlation, except where the two are too close to separate; asterisks mark p < 0.01.

Doing more with fewer questions

BenchMIRT can also help identify which questions in an evaluation are most informative about the capability the benchmark is trying to measure.

Using BenchMIRT’s question-level estimates, we ranked questions across the same 16 benchmarks used to train BenchMIRT and kept those that did the best job of distinguishing stronger from weaker models, while still preserving a mix of easier and harder questions.

Across those benchmarks, keeping only 10% of the questions generally preserved nearly the same picture of which models were stronger or weaker on the underlying safety or reasoning capability as using the full set. Keeping 50% of the questions often matched the full benchmark’s measure of those capabilities even more closely.

BenchMIRT can also use the patterns it learns across models and questions to predict how a model would perform on a benchmark question it hasn’t been observed answering. In our experiments, it correctly predicted whether a model would answer a held-out question correctly 79% of the time. By comparison, a simpler approach that assumes a model will perform on each question about as well as it does on the benchmark overall was correct 70% of the time.

In practice, that means BenchMIRT can estimate model performance more precisely from what it has already learned about the model’s abilities and the demands of each question, without needing to evaluate every model on every question.

What this could mean for LLM evaluation

BenchMIRT offers a way to better understand and refine the benchmarks researchers use to evaluate model capabilities. By looking at individual questions rather than only overall scores, it can reveal when a benchmark mixes together different capabilities, identify clusters of questions that behave differently from the rest, and surface questions that add little useful information about the capability the benchmark is meant to measure.