Teaching AI to read a map

Ask an AI model what's in a photo of a mall and it'll nail it: food court, escalators, that one kiosk selling phone cases. Ask it to draw the route from the entrance to the reptile house at a zoo, though, and things fall apart fast. Google Research's new work, called MapTrace, digs into exactly why multimodal models are so bad at this and, more usefully, how to fix it.

The root problem is data, not architecture. Models learn that the word "path" goes with pictures of sidewalks, but nobody ever teaches them the actual rules of navigation — that a route can't cut through a wall, that it has to be a connected sequence of points, that gift shops are not shortcuts. Hand-labeling millions of paths across mall maps, museum floor plans, and theme park layouts would take forever, and a lot of the best map imagery is proprietary anyway. So the usual fix — more labeled data — was basically off the table.

Google's answer was to generate the data synthetically, using AI to both create and grade its own homework. An LLM dreams up map prompts (a zoo with interconnected habitats, a mall with a central food court), a text-to-image model renders them, and then a series of automated critics take over. One MLLM, dubbed the Mask Critic, checks whether the shaded regions in a generated map actually represent walkable space rather than random blobs of similar color. Once a clean walkable mask exists, the team converts it into a graph — nodes and edges, like a road network — runs Dijkstra's algorithm to compute genuinely shortest paths, and then hands the result to a second critic, the Path Critic, which vetoes anything that looks unrealistic. The output: 2 million annotated map-and-path examples, now open-sourced.

The payoff showed up when they fine-tuned models on just 23,000 of those paths and tested them on MapBench, a benchmark of real maps the models had never seen. Gemini 2.5 Flash's path-tracing error, measured by a metric called NDTW, dropped from 1.29 to 0.87 — the best score in the study. Gemma 3 27B improved too, and its success rate — the share of attempts that produced any valid, parsable path at all — jumped by 6.4 points. That reliability gain matters as much as the accuracy gain: a model that fails outright half the time is far less useful than one that's occasionally imprecise but consistently produces something usable.

The critics aren't flawless — manual review found the Path Critic was right about 76% of the time and the Mask Critic about 83%, with both occasionally mistaking background color for pathways or missing narrow routes. Google frames the whole exercise as evidence that spatial reasoning isn't something these models pick up for free from internet-scale pretraining; it has to be explicitly taught, even if the teaching material is synthetic. The stated ambition — indoor robots reading floor plans, accessibility tools describing routes for blind users, smarter transit apps — is the obvious next step once a model can reliably tell a hallway from a wall.