Sensible Agent: A framework for unobtrusive interaction with proactive AR agents

Google's Project Astra showed off what proactive AR agents could look like, but it left one big problem unsolved: you still have to talk to the thing. Say a command out loud in a museum, a crowded subway car, or mid-workout at the gym, and you either feel awkward or you simply can't. Google Research's new framework, Sensible Agent, presented at UIST 2025, tries to fix that by rethinking not just what an assistant suggests, but how it interacts with you at all.

The system splits the problem into two questions running in parallel: what does the user need right now, and what's the least disruptive way to offer it. A context parser reads the camera feed with a vision-language model and checks ambient noise with YAMNet, an audio classifier, to figure out whether you're cooking, riding a train, or standing in a grocery aisle. From there, a query generator using chain-of-thought reasoning and six examples from an internal study decides on an action — recommend a dish, show a shopping list, translate a sign — and picks a format: multi-choice, binary, or a simple icon. If your hands are full, it might ask for a head nod instead of a spoken yes. If the room is loud, it shows a visual cue rather than speaking.

Google tested this against a voice-only baseline modeled on Project Astra, with 10 participants running through 12 scenarios each, split between 360-degree video simulations and physically staged environments covering restaurants, transit, museums, gyms, grocery stores, and kitchens. The numbers are stark: on the NASA Task Load Index, mental demand scored 21.1 for Sensible Agent versus 65.0 for the voice baseline, a gap that held up statistically (p < .001). Perceived effort dropped just as sharply. Usability scores came out roughly even between the two systems, but preference didn't — participants rated Sensible Agent 6.0 out of 7, against 3.8 for the baseline.

There's a catch, and Google doesn't bury it. Sensible Agent took longer per interaction, 28.5 seconds on average versus 16.4 for voice commands, because the agent proposes first and waits for confirmation rather than executing immediately. Users apparently didn't mind the extra few seconds if it meant not talking to their glasses in public. More interesting is the qualitative shift researchers noticed: people described the proactive, gesture-based version less as a tool and more as a partner that read the room, which is a different kind of trust than just getting a fast answer.

Google frames this as groundwork rather than a finished product, with plans to add long-term personalization, cross-device support, and on-device inference for privacy, plus eventual crossover into smart homes and robotics. The prototype currently runs on Android XR and WebXR, built by a large team spanning multiple Google research groups. It's a narrow academic study — 10 people, six scenario types — but the direction is a useful corrective to the current AR assistant pitch, which has mostly assumed everyone's fine narrating their day out loud to their glasses.