XR Blocks: Accelerating AI + XR innovation

Google has a habit of open-sourcing the plumbing before anyone else builds the house, and XR Blocks fits that pattern neatly. Presented at ACM UIST 2025, it's a cross-platform framework that tries to do for AI-plus-XR development what PyTorch and JAX already did for machine learning: give creators a mature set of tools instead of forcing them to hand-wire cameras, depth sensors, and language models every single time.

The gap Google is targeting is real. Building something like an AR assistant that recognizes objects, understands context, and responds intelligently currently means stitching together perception systems, rendering engines, and AI APIs with no shared vocabulary. XR Blocks proposes a "Reality Model" built around six primitives — user, physical world, virtual interface, context, AI agents, and social entities — all orchestrated through something they call a Script. The pitch is that a developer should be able to write "when the user pinches an object, generate a poem about it" and have that read almost like actual working code, not pseudocode that still needs translating into a mess of API calls.

Under the hood it's built on familiar, accessible tech: WebXR, three.js, LiteRT, and Gemini for the AI layer. That's a deliberate choice. Google explicitly says it favors pragmatism over completeness, arguing that a framework trying to be a perfect, comprehensive solution for a fast-moving field like AI and XR would be obsolete the moment it shipped. So instead of some monolithic system, they built modular, swappable components — camera and depth modules, an ai module with simple `.query` and `.runModel` calls, and UX helpers like `.selectable` and `.draggable` — running on both desktop and Android XR hardware.

The project isn't theoretical. It already underpins Sensible Agent, a proactive AR assistant also presented at UIST 2025, and XR-Objects, a system that turns everyday physical items into interactive digital objects with virtual buttons attached. Google frames these as proof that researchers can skip the low-level grind and focus on harder questions, like how humans and AI agents should actually collaborate in a shared space. The whole thing ships with open-source templates, demos, and code on GitHub, which is really the tell here: Google wants a community building on this before the AI-XR space consolidates around someone else's stack.

Whether XR Blocks becomes the default toolkit or just another abandoned Google research repo depends entirely on adoption outside Mountain View, something the company's track record with dev tools makes genuinely uncertain.