Video generation models as world simulators
OpenAI Blog
OpenAI trained Sora, a large-scale video generation model using diffusion and transformers on variable-length videos and images. The model can generate up to 60 seconds of high-fidelity video from text descriptions. The researchers propose that scaling video generation represents a viable approach to creating general-purpose simulators of physical phenomena.
Why it matters
We explore large-scale training of generative models on video data. Specifically, we train text-conditional diffusion models jointly on videos and images of variable durations, resolutions and aspect ratios. We leverage a transformer architecture that operates on spacetime patches of video and image latent codes. Our largest model, Sora, is capable of generating a minute of high fidelity video. Our results suggest that scaling video generation models is a promising path towards building general purpose simulators of the physical world.
