GH-ESD: Grounded Hypothesis-Driven Error Slice Discovery for Instance-Level Vision Tasks
Apple
Apple researchers built a way to find exactly where vision models fail, not just that they fail. It works on object detection and segmentation, not just image labeling.
Based on reporting by Apple — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Vision models trip up in patterns, not just random noise. Researchers have long tried to catch these patterns — called error slices — by clustering similar images or checking against a list of predefined attributes. That works fine when you're asking a model to label a whole photo. It falls apart when the task is object detection or segmentation, where a single image can contain a dozen instances, and the failure has more to do with spatial relationships or context than with the image as a whole.
A team at Apple, including Wei Zhang and Chaoqun Wang as equal contributors alongside Zixuan Guan, Ping Sheng Kao, Pengfei Zhao, Peng Wu and Sifeng He, built something called GH-ESD to handle this gap. Short for Grounded Hypothesis-Driven Error Slice Discovery, it flips the usual approach on its head. Instead of hunting for clusters after the fact, it generates hypotheses first — using large language model priors combined with grounded visual evidence — about what kind of relational or spatial pattern might be causing failures. Then it uses vision language models to hunt for those hypothesized slices at the level of individual instances, not whole images. Finally, it checks its own work with statistical trend analysis over the instance-level errors, so a hypothesis has to earn its place rather than just sound plausible.
To actually test this, the team also built GESD, the Grounded Error Slice Dataset — a benchmark with expert-defined, spatially grounded slices pulled from real detection and segmentation failures. That matters because instance-level slice discovery hasn't really had a proper benchmark before; most existing tools were designed and measured for classification, leaving detection and segmentation as something of an afterthought.
The numbers back up the approach. On the GESD benchmark, GH-ESD lifted Precision@10 to 0.73 from a baseline of 0.63 on detection tasks — a solid ten-point jump — and the framework held up for segmentation scenarios too. Beyond the metric, the authors say the slices it surfaces are interpretable enough to point toward concrete fixes, rather than just flagging that something, somewhere, is wrong.
My take — AI-written commentary, not fact-checked reporting
This is the unglamorous plumbing work that actually matters more than another leaderboard-topping model release — knowing why a detector fails on certain instances is worth more than knowing it failed at all. Building a dedicated benchmark for instance-level slices is the quietly useful part here, since half the field's tools were never designed for anything beyond simple classification. Expect this kind of hypothesis-driven debugging to matter a lot more as detection and segmentation systems get deployed somewhere the stakes are higher than a demo.
Read more about this at: Apple