Evaluation & Hallucination Detection for Abstractive Summaries
Eugene Yan
Researchers discuss methods for evaluating abstractive summaries and detecting hallucinations, identifying four key dimensions: fluency, coherence, relevance, and consistency. Studies found hallucination rates ranging from 30% to 92% depending on the dataset, with consistency being the most objective dimension to measure automatically. The field is moving away from reference-based metrics toward context-based and LLM-based evaluation approaches due to the high cost and often poor quality of human-written reference summaries.
Why it matters
Reference, context, and preference-based metrics, self-consistency, and catching hallucinations.