D4RT: Teaching AI to see the world in four dimensions

Google DeepMind just put out a model called D4RT that tries to solve a problem most of us never think about: how do you take a flat video and figure out where everything actually is in space, and how it's all moving, at any given moment? Humans do this without effort. Computers have historically needed a pile of separate specialized systems stitched together — one for depth, one for tracking, one for camera pose — and the whole pipeline was slow and prone to falling apart when things got complicated, like an object walking behind a pole and popping out the other side.

D4RT ditches that patchwork approach for a single encoder-decoder Transformer. The encoder digests the video into a compressed representation of geometry and motion, and then a lightweight decoder answers a very specific question over and over: where is this pixel, at this time, from this camera angle? Because each of those queries is independent, they run in parallel on modern chips, which is really the whole trick here. Instead of computing everything about a scene up front, the model only computes what's actually asked for.

The efficiency numbers are the headline. DeepMind says D4RT runs 18x to 300x faster than prior state-of-the-art methods, and in one test it chewed through a full minute of video in about five seconds on a single TPU — a job that used to take up to ten minutes. That's not a marginal speedup, that's the difference between a research demo and something you could plausibly run on a robot or a pair of AR glasses in real time.

What you get for that speed is a model that handles point tracking, full 3D point cloud reconstruction, and camera pose estimation all through the same querying interface, rather than three separate tools duct-taped together. And the paper claims it holds up better on moving objects specifically, where older methods tend to duplicate things or just lose track of them entirely. Google frames this as a step toward AI systems with something like a genuine internal model of physical reality — separating what the camera is doing from what objects in the scene are doing — which is the same language showing up around projects like Genie 3 and Gemini Robotics.

Whether D4RT actually becomes the backbone for shipped AR hardware or warehouse robots is a separate question from whether the architecture is clever, and right now this is a research release with benchmark numbers, not a product. But the direction is clear enough: DeepMind is betting that unifying 4D perception into one flexible query system beats maintaining a zoo of specialist models, and the speed gains here make that bet look pretty reasonable.