When LLM judges agree, should we believe them?
Amazon Science ● Covered by 2 sources
Researchers propose a dependence-aware way to aggregate outputs from multiple LLM judges so that agreement counts don’t overstate independent evidence when judges share biases. The method’s strongest results using 10 judge models at temperature 0 reach 0.912 accuracy for relevance, up from 0.820 with weighted majority vote and 0.804 with uniform majority vote. As a result, evaluation scores adjust for learned pairwise correlation between judges rather than assuming independent errors, improving accuracy across three binary tasks.
Why it matters
Discounting the opinions of LLM judges with highly correlated outputs ensures that panels of judges reflect a true diversity of perspectives.