GH-ESD: Grounded Hypothesis-Driven Error Slice Discovery for Instance-Level Vision Tasks
Apple ML Research
Researchers developed GH-ESD, a framework for discovering error slices in vision models that uses language model priors and vision-language models to identify systematic failures in instance-level tasks like object detection and segmentation. The method achieved Precision@10 of 0.73 versus 0.63 for baseline approaches on a new GESD benchmark for detection tasks. This approach enables identification of interpretable, spatially grounded failure patterns that can guide targeted model improvements beyond existing attribute-based slice discovery methods.
Why it matters
Systematic failures of vision models on semantically coherent subsets, known as error slices, reveal limitations in robustness and evaluation. Existing slice discovery approaches largely model slices as clusters in representation space or combinations of predefined attributes. While effective for image-level classification, such formulations are insufficient for instance-level tasks such as object detection and segmentation, where failures often arise from contextual relational and spatially grounded visual patterns. We propose GH-ESD (Grounded Hypothesis-Driven Error Slice Discovery), a…
