Rethinking Evaluation in Retrieval-Augmented Personalized Dialogue: A Cognitive and Linguistic Perspective

arXiv cs.CL · 18 hours ago

Researchers examined evaluation methods for retrieval-augmented personalized dialogue systems, finding that surface-level metrics like BLEU and ROUGE fail to capture conversational quality issues such as contradictions and incoherence. Human evaluators and LLM-based judges showed close alignment with each other but diverged significantly from lexical similarity metrics. The study argues for cognitively grounded evaluation frameworks that better reflect principles of natural human communication rather than lexical similarity measures.