Meet ‘Code-as-World’: An Agentic Loop That Rewrites Real Videos Into Executable MuJoCo Physics Programs
MarkTechPost Michal Sutter
MirroS turned real videos into editable MuJoCo scene files. It’s a physics-first bet: the model can verify what happened, not just guess the next frame.
Based on reporting by MarkTechPost, Michal Sutter — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
MirroS has released Code-as-World, a system built on a simple claim: video is evidence of a physical scene, not the scene itself. Instead of treating pixels, captions, or latent vectors as the final answer, it turns footage into executable world representations that MuJoCo can run, inspect, and re-simulate. That makes the output more than a description. It becomes something an agent can test against the source video and then edit if the fit is wrong.
The core object is a scene.json built from three parts: composition, evolution, and appearance. Composition covers objects, geometry, dimensions, mass, friction, and gravity. Evolution handles the initial state, forces, contacts, collisions, duration, and termination. Appearance covers camera, lighting, materials, background, and render settings. Change the appearance and the physics stays the same. Change the composition and you change what the world can do.
To recover those executable worlds from video, the system does not rely on one-shot prediction. It runs a propose → instantiate → execute → render → verify loop for up to five rounds. On the video side, it uses SAM 3 for instance masks and image-plane tracks, VGGT-Omega for depth and camera geometry, and SAM 3D for per-object meshes. Candidate rollouts are projected back into the source view and checked against key frames using RGB, depth, masks, and trajectories. If the budget runs out and the hypothesis still fails, the system rejects it.
That loop matters because the verified worlds become training supervision. MirroS says the worlds provide exact physical labels that raw video does not carry. Phase 1 uses supervised fine-tuning on 73,335 image-space QA pairs from RefCOCO/+/g, RefCLEF, and GOT-10K. Phase 2 uses GRPO on world-space VQA built from 1,585 text-driven and 988 video-driven executable worlds. Training ran on eight NVIDIA H100 GPUs.
The results are solid, and specific. On QuantiPhy-validation, the 9B model reaches 55.4 MRA, ahead of Gemini-3.1 Flash at 54.8 and well above the strongest open-weight baseline, Qwen3-VL-32B-Instruct at 40.2. The 4B model scores 50.6, and the 27B reasoning model reaches 58.6. MirroS also says the five-round search beats Best-of-5 independent sampling at the same compute budget on Visual Alignment, Object IoU, Traj-ADE, and Accuracy@2%D, and that the gain holds under both execution engines.
The release is practical too, at least for research and internal prototypes. MirroS has put out the GitHub repo and two Apache 2.0 checkpoints, Code-as-World-VL-4B and Code-as-World-VL-9B, both fine-tuned from Qwen3.5-4B and Qwen3.5-9B. They’re BF16 safetensors served through vLLM behind an OpenAI-compatible /v1 endpoint, with 16 sampled frames per video and a max model length of 4608.
My take — AI-written commentary, not fact-checked reporting
This is the right kind of AI paper: less chatbot theater, more machinery. Open weights under Apache 2.0 help too, because a physics model that can’t be checked is just a very confident narrator in a lab coat. The bigger lesson is plain enough: if a system claims to understand the physical world, make it write down the world in code and see if reality agrees.
Read more about this at: MarkTechPost