Introducing ConTextual: How well can your Multimodal model jointly reason over text and image in text-rich scenes?

Reading a subway map or getting a joke in a meme takes more than spotting words or objects separately, it takes both at once, fused into one understanding. That's exactly what a new benchmark from UCLA researchers is designed to expose as a blind spot in today's top multimodal models.

Called ConTextual, the dataset throws 506 tricky prompts at models across eight everyday scenarios: reading clocks, shopping, navigating, parsing infographics, using mobile apps and webpages, plus abstract scenes like memes and natural photos. Each sample pairs a text-heavy image with a human-written instruction and a human-written reference answer, and a GPT-4 judge decides whether a model's response is acceptable by comparing it against that reference.

The results aren't flattering. Thirteen models were tested, spanning augmented setups (GPT-4 fed OCR text and captions instead of raw pixels), closed systems like GPT-4V and Gemini-Vision-Pro, and open models such as LLaVA-1.5-13B, Qwen-VL-7B and Idefics-9B. GPT-4V came out on top overall, even beating humans on abstract reasoning tasks involving memes and quotes, likely because it's seen plenty of that kind of content during training. But it fell apart on something as basic as reading time from a clock or infographic, a task where humans breeze through without effort.

Open-source models fared worse across the board, doing fine on natural scenes and abstract images but collapsing on navigation, shopping, and mobile app screenshots, suggesting those categories barely showed up in their training data. The augmented approach, bolting a language model onto OCR output and captions, was the weakest of all, with humans approving only 17.2% of its answers. That gap makes a simple point: converting an image to text and reasoning over the text loses too much of what actually matters, the spatial and visual relationship between words and objects.

The UCLA team, led by researchers Rohan and Hritik alongside Kai-Wei Chang and Nanyun Peng, has opened a leaderboard and is accepting submissions on both a 100-sample validation set and the full 506-sample test set. Their diagnosis for what's missing is straightforward: better image encoders, sharper image descriptions, and finer alignment between vision and language, the kind of unglamorous engineering work that rarely makes headlines but determines whether an AI assistant can actually help someone find their gate at an airport instead of just describing that there's a sign.