Out-of-Domain Finetuning to Bootstrap Hallucination Detection

Eugene Yan · 2 years ago · 35

Researchers demonstrated that pre-finetuning a model on out-of-domain Wikipedia summaries before finetuning on the Factual Inconsistency Benchmark (FIB) dataset significantly improves hallucination detection in news summaries. The approach achieved a PR AUC of 0.85, representing a 23% improvement over finetuning on FIB data alone. This finding suggests that transfer learning through multi-stage finetuning can reduce the need for large amounts of task-specific labeled data by leveraging related datasets from different domains.