Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering
Apple Machine Learning Research
The article proposes a new way to evaluate video caption quality for Visual Large Language Models that avoids penalizing valid differences in wording or focus. It measures caption quality by running multiple-choice question answering rather than relying on overlap with ground-truth references. As a result, evaluation is reframed around information fidelity and aims to give a more fine-grained quality analysis than existing one-dimensional metrics.
Why it matters
Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This paradigm suffers from the “one-to-many” nature of video description, where high-quality captions are often penalized for lexical mismatches or valid shifts in visual focus. Furthermore, such assessments are typically one-dimensional, failing to provide a fine-grained analysis of caption quality. To address this, we redefine caption quality via information fidelity: A caption must maximize the coverage…