Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space Richard MacManus
Runway just showed WorldPrompt, a way to steer its new real-time video worlds. It’s the closest the company has come to making interactive scenes feel like software, not clips.
Based on reporting by Latent Space, Richard MacManus — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Runway has a new research preview called GWM Worlds 2, and the headline feature is WorldPrompt: a way to describe a world, lock in parts of it — even the first frame — and then feed in timestamped actions as the scene unfolds. The pitch is simple enough. Instead of generating a finished video, the model is supposed to keep a world alive in real time, with video and audio moving together.
That’s a very different job from the usual text-to-video demo. Runway says GWM Worlds 2 streams continuous 720p video at 24 frames per second and audio at 48,000 Hz. Under the hood, the company starts with its audio-video foundation model, fine-tunes it for WorldPrompt, then post-trains it to generate autoregressively. After that comes distillation, which is how Runway says it gets the system fast enough to run in real time.
The engineering problem is ugly. Autoregressive generation means the model keeps feeding its own output back in, frame after frame, so small mistakes can pile up. Runway’s co-CEO Anastasis Germanidis said the system can be pushed from around 50 denoising steps to four, but that kind of shortcut comes with quality tradeoffs. Kamil Sindi, Runway’s CTO, said long runs also create memory and context problems: the model has to decide what to keep and what to throw away without blowing up GPU memory. And the model still does not have perfect memory, according to Robin Kahlow, Runway’s principal research scientist.
The company is also careful about what WorldPrompt is not. It is prompting, not scripting. So unlike Minecraft or Roblox, GWM Worlds 2 does not give users a proper state machine or a programming language to control. Kahlow said the model can handle movement fairly reliably, but more difficult instructions still cause flaws. The whole thing is a research preview, not a polished product, and Runway clearly knows the gap between a promptable scene and a truly programmable world.
Still, the target is bigger than games. Kahlow pointed to robotics simulation and to testing agents at scale, where thousands of simulated environments would be easier if the model could generate them on demand. There’s no structured state for the agent to read, only the same kind of camera view it would get in the real world. That’s the point, and also the inconvenience. Runway is betting that if you can make generated worlds coherent enough, they become useful for more than watching. They become places to test, train, and maybe eventually act.
My take — AI-written commentary, not fact-checked reporting
Runway is doing the unglamorous part of the world-model story: memory, latency, and error accumulation, not just shiny demo clips. That’s the right obsession. The industry has spent years pretending that more pixels automatically mean more intelligence, which is a lovely way to burn GPU hours and call it progress.
Read more about this at: Latent Space