Show Me Examples: Inferring Visual Concepts from Image Sets
Apple ML Research
Researchers introduced VICIS, a task that evaluates whether vision-language models can infer shared visual concepts from sets of example images and apply them to new inputs. State-of-the-art VLMs performed poorly on this benchmark, often ignoring visual context or producing biased outputs. The authors developed a training framework and architecture that learns to infer visual concepts from image sets, demonstrating improved accuracy and generalization to unseen concepts and modalities like sketches on ImageNet and WordNet data.
Why it matters
Vision-language models (VLMs) can follow complex textual instructions, yet they struggle to reason from purely visual context. In particular, current models fail to infer shared concepts from sets of example images and apply them to new inputs. We introduce Visual Concept Inference from Sets (VICIS), a task that evaluates this capability. Given a small context set of images sharing a concept and a query image, the model must generate new images that preserve the context-defined concept while remaining consistent with the query. We show that state-of-the-art VLMs perform poorly on this task…
