Into the Omniverse: How Developers Turn Ideas Into Simulations With Frontier AI Agents

NVIDIA is using frontier AI models as build partners for simulation work, not just chat toys. Across a set of projects, developers gave natural-language instructions to agents, reviewed what came back, and then pushed the work forward with corrections. The common thread is Omniverse libraries: GPU-accelerated physics, rendering, and sensor simulation that give the agents something real to assemble against.

Frank DeLise, an Omniverse product manager at NVIDIA, used Astra to turn a SimReady warehouse and humanoid robot into an interactive simulator with first- and third-person views. He had Astra connect the physics, scene update, rendering, and UI pieces, then generate the animation and application code to tie it all together. The point was practical: before automating warehouse work, developers need to explore behavior in an interactive environment, not just stare at a diagram.

The same pattern shows up in autonomous driving. Doyub Kim asked Astra to build Zero to Alpamayo, a reusable simulation environment based on San Francisco’s Market Street. He had the workflow connected in stages, with checks after each integration, so changes to the scene, sensor, or driving model could be traced through the system. A separate Cosmos3-Nano experiment changed weather and lighting in recorded simulation videos so Kim could compare how the driving model reacted to the same scenario under different conditions.

Then there’s sensor validation, where the work gets less flashy and more useful. Ashley Reid directed Astra and Claude Fable 5 agents to compare ovrtx camera and raw LiDAR outputs with recorded data. Over about three days, the agents created two digital twins from scratch and improved two existing ones, fixing missing objects, geometry, and materials when the camera and LiDAR metrics said the scene was off. That is the sort of feedback loop simulation teams actually need.

Other examples go from robotics sports to disassembly to the browser. Tae Kim used sports videos and instructions to get Astra building Robo Olympics, where a Unitree G1 humanoid was tested on sports motions and cleared a single hurdle in 64 of 100 trials. Jens Jebens had Astra model a car suspension in PTC Onshape and configure it in Isaac Sim so a robot could reach the bolts. Nic Johns built an International Space Station browser app from NASA assets and telemetry with one prompt, then nudged the scene so Earth showed on the daytime side. NVIDIA’s pitch is simple: let the agent do the assembly, keep a human in the loop, and use simulation as the proving ground before the real world does the rejecting.