Diffusion Models for Video Generation

Diffusion models had a great run making still images look almost suspiciously good. Now researchers are pointing the same machinery at video, and the jump is not a small one. Lilian Weng's latest writeup frames video generation as a strict superset of image generation—after all, a single image is just a one-frame video—but the practical gap between the two is enormous.

The core problem is temporal consistency. A model can nail a gorgeous single frame and still fall apart the moment it has to keep that frame's objects, lighting, and physics stable across dozens of subsequent frames. That requirement forces these systems to encode something closer to a working model of how the physical world behaves, not just how pixels are typically arranged. Getting a diffusion model to hallucinate a convincing cat is one thing; getting it to hallucinate a cat that moves believably for two seconds is another problem entirely.

Data scarcity makes this worse. Text-image pairs are already plentiful thanks to a decade of web scraping and captioning work. Text-video pairs, at the scale and quality needed to train these models, are far rarer and far more expensive to curate. High-dimensional video data is heavier to store, harder to label consistently, and there's simply less of it floating around the internet in clean, usable form compared to static images.

Weng treats this as a direct extension of her earlier explainer on diffusion models for images, essentially saying: understand the fundamentals first, because video builds on every one of those mechanics while adding a time axis that breaks a lot of assumptions. That framing matters. It signals that video diffusion isn't a separate research track bolted onto image diffusion—it's the same theoretical foundation being stretched to handle a much less forgiving kind of data.