DeepAmbigQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness

Apple Machine Learning Research · 2 months ago · 15

Researchers introduced DeepAmbigQA, a dataset of 3,600 questions designed to test whether large language models can handle ambiguous entity names and multi-step reasoning in open-domain question answering. State-of-the-art models including GPT-4 achieved only 0.13 exact match on ambiguous questions and 0.21 on non-ambiguous ones, revealing significant gaps in answer completeness. The benchmark exposes the need for QA systems that better gather evidence across multiple entities and disambiguate between similarly named entities.