Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)
Eugene Yan
Researchers evaluated large language models used as automated judges to assess the quality of other LLM outputs, comparing approaches like direct scoring, pairwise comparisons, and reference-based evaluation. Studies show LLM-evaluators achieve correlation with humans ranging from 0.3 to 0.9 depending on the task, with larger models (52B+ parameters) approaching performance of finetuned preference models when using chain-of-thought prompting. Organizations can use LLM-evaluators to scale evaluation beyond human annotation while choosing between classification metrics or correlation metrics depending on whether they need binary outputs or ranked assessments.
Why it matters
Use cases, techniques, alignment, finetuning, and critiques against LLM-evaluators.
