AgentHands: Generating interactive hand gestures for spatially grounded agent conversations in XR

Google Research has a new answer to a very specific XR problem: once an assistant leaves the flat screen, how should it point, warn, and demonstrate in three dimensions? The company’s prototype, AgentHands, was published at CHI 2026 and is built to give AI agents co-speech hand gestures that line up with their spoken words inside XR.

The idea starts with a simple observation: hands do a lot more than point. In AgentHands, gestures can show shape, mimic an action, or add social cues, and they can do it in a way that is tied to the user’s surroundings. Google says it first ran a formative study with XR and HCI experts to figure out what makes a virtual hand readable in 3D, then turned those findings into a taxonomy covering handedness, spatial placement, and timing plus visual effects.

That taxonomy feeds a workflow built to convert an LLM’s response into motion. Users can tag objects in their environment with eye gaze and scene reconstruction, creating 3D bounding boxes the agent can reference. A gesture library groups behaviors into deictic, iconic, and expressive categories. The model then inserts inline GestureEvents into its response, and a local parser on the headset synchronizes text-to-speech with the animation engine using word-level timestamps.

The demos are aimed at tasks that benefit from being shown, not just described. In an orchid-care scenario, the agent moves its hands to the plant’s base and traces the air roots. In a 3D printer walkthrough, it can show the turn-and-click sequence for knobs and file selection. Google also describes a wellness-coach style interaction where the agent uses a warning gesture and a visual effect to flag unhealthy behavior.

The user study was small — 12 people — but the pattern was clear. Compared with a speech-only baseline using the same scripted content, AgentHands made it easier to find objects and follow directions, and users rated the required activities as easier to understand. Participants also reacted strongly to safety cues, including a burn effect used for a hot nozzle, and said the hands felt like a partner guiding them.