From Preferences to Principles: Rubric-Based Alignment for Grounded Knowledge Answers
Apple Machine Learning Research
The authors propose a rubric-based reward framework that generates query-specific, evidence-grounded rubrics and uses them for post-training to supervise open-domain question answering. Averaged across three evaluation axes (composition, grounding, and instruction-following), it improves 6.5% over an instruction-tuned baseline. The approach changes training by replacing a single scalar reward with multi-dimensional rubrics conditioned on retrieved evidence, yielding more factual support and better coherence and instruction adherence.
Why it matters
Designing effective reward signals for open-domain question answering is challenging because high-quality responses must simultaneously satisfy multiple aspects of answer quality that are difficult to capture with a holistic scalar objective. We introduce a rubric-based reward framework that generates query-specific rubrics grounded in retrieved evidence and decomposed into multiple quality dimensions, providing fine-grained supervision during post-training. Averaged across three evaluation axes (composition, grounding, and instruction-following), our approach improves over the…