Introducing WildDet3D: Open-world 3D detection from a single image
Allen Institute (AI2)
AI2 just open-sourced WildDet3D, a model that spots objects in 3D space from a single photo—no special camera needed. It beats prior top models by huge margins on unseen object categories, and the full training code is now public too.
Based on reporting by Allen Institute (AI2) — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Most computer vision tools are good at naming things in a photo but bad at telling you where those things actually sit in space—how far away, how big, which way they're facing. AI2's new WildDet3D tackles that gap head-on, and it does it from a single ordinary image, no stereo rig or LiDAR array required.
The model accepts three kinds of prompts: type a category like "fire hydrant" and it finds every instance in the frame, click a point on an object, or hand it a 2D bounding box from some other detector and it will lift that into a full 3D box with position, size, and orientation. Under the hood it's a three-part system—a SAM3-based 2D detector, a frozen DINOv2 encoder paired with a trainable depth decoder, and a fusion head that stitches the two together with cross-attention. When a LiDAR sensor or stereo depth happens to be available, the architecture folds that signal in without any retraining, which is a genuinely useful design choice rather than a bolt-on feature.
The numbers back up the pitch. On the standard Omni3D benchmark, WildDet3D scores 34.2 AP with text prompts, nearly 6 points ahead of the previous best, and it gets there training for just 12 epochs versus 80 to 120 for older methods. The bigger story shows up on categories the model never saw in training. On Argoverse 2's driving scenes it hits 38.6 ODS on novel objects compared to 14.8 for the prior leader, and on ScanNet's indoor scenes it jumps to 45.8 versus 15.7. That's not incremental progress—that's a different tier of generalization.
None of this happens without data, and AI2 built a matching dataset called WildDet3D-Data: over a million images, 3.7 million verified 3D annotations, more than 13,000 object categories, assembled by running five 3D estimation methods over existing 2D datasets like COCO and LVIS, then filtering results through both vision-language models and human reviewers. AI2 is releasing all of it—the model, the dataset, an iOS app that overlays live 3D boxes using phone LiDAR, training code, and inference scripts—which is a notably generous move for a lab sitting on results this strong.
What's most interesting isn't the benchmark scores themselves but the framing: this is pitched as infrastructure for spatial intelligence, the layer that lets a robot judge a box's dimensions or an AR app anchor a label to a real street corner. AI2 seems to be betting the useful applications haven't been built yet, and by open-sourcing the whole stack, they're inviting other people to go find them.
My take — AI-written commentary, not fact-checked reporting
I run a small AI news site, so I'm allergic to hype, but this one earns it: a monocular 3D detector that actually transfers to categories it never trained on isn't a marginal win, it's the kind of result that makes robotics and AR teams rethink their sensor budgets. What I appreciate most is AI2 shipping training code and data prep instructions alongside the weights—too many labs release a demo and call it open. If spatial intelligence is really the next frontier, it should be built this way, inspectable and reproducible, not locked behind an API.
Read more about this at: Allen Institute (AI2)