Evaluation & Hallucination Detection for Abstractive Summaries

Eugene Yan · 3 years ago · 46

Researchers discuss methods for evaluating abstractive summaries and detecting hallucinations, identifying four key dimensions: fluency, coherence, relevance, and consistency. Studies found hallucination rates ranging from 30% to 92% depending on the dataset, with consistency being the most objective dimension to measure automatically. The field is moving away from reference-based metrics toward context-based and LLM-based evaluation approaches due to the high cost and often poor quality of human-written reference summaries.