AgentHands: Generating interactive hand gestures for spatially grounded agent conversations in XR
Google Research 1 month ago 11
AgentHands, a CHI 2026 research prototype, generates spatially grounded hand gestures in XR by mapping LLM responses to synchronized 3D co-speech motion tied to word timestamps and headset scene understanding. The prototype was evaluated in a within-subjects study with N = 12 participants versus a speech-only baseline, showing significant gains in spatial grounding (p < 0.05). As a result, XR instructions become easier to locate, follow, and notice for spatial and safety cues, with reduced cognitive load from embodied, timed gestures.