StreetReaderAI: Towards making street view accessible via context-aware multimodal AI
Google Research
Google built an AI tool that lets blind users explore Street View by voice, describing streets, crossings, and landmarks in real time. It turns a purely visual product into something screen reader users can actually navigate and ask questions about.
Based on reporting by Google Research — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Google Street View has always been a visual medium in the strictest sense: 220 billion images across more than 110 countries, and not one pixel of it accessible to someone using a screen reader. That's the gap Google Research is trying to close with StreetReaderAI, a prototype detailed in a paper presented at UIST'25 that pairs Gemini's multimodal reasoning with a rebuilt, keyboard-and-voice navigation layer.
The mechanics are straightforward once you see them. Arrow keys pan the view and StreetReaderAI narrates the heading aloud — "Now facing: Northeast" — while also flagging whether there's a landmark in front of you or room to keep walking. Pressing up or down triggers a "virtual step," with the system reporting distance traveled and nearby points of interest as you go. Two Gemini-backed subsystems do the heavy lifting: AI Describer, which narrates a scene either in safety-focused mode or as a tour guide dropping in historical trivia, and AI Chat, which holds a running memory of everything you've seen in a session — up to 1,048,576 tokens, or roughly 4,000 images worth of context — so you can ask something like
Read more about this at: Google Research