When LLM judges agree, should we believe them?
Amazon Science 1 month ago 28 ● 2 sources
Researchers propose a dependence-aware way to aggregate outputs from multiple LLM judges so that agreement counts don’t overstate independent evidence when judges share biases. The method’s strongest results using 10 judge models at temperature 0 reach 0.912 accuracy for relevance, up from 0.820 with weighted majority vote and 0.804 with uniform majority vote. As a result, evaluation scores adjust for learned pairwise correlation between judges rather than assuming independent errors, improving accuracy across three binary tasks.
