The Sequence Knowledge - Issue 911: Distilling Diffusion and Multimodal Models
Substack Jesus Rodriguez
Text distillation is simple: a smaller model learns to copy a bigger one. Diffusion and multimodal models make that much harder because the target is a whole process, not just an answer.
Based on reporting by Substack, Jesus Rodriguez — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Text distillation is easy to explain because the goal is so clean. A large language model gets a prompt, produces a response or a next-token distribution, and a smaller model is trained to imitate that output. The teacher says one thing. The student learns to say the same thing, or something close enough to pass for it.
Diffusion distillation is messier. A diffusion model doesn’t spit out an image in a single forward pass. It begins with noise and keeps revising that noise until a coherent sample emerges. What matters is the path as much as the result. The model is not just answering a query; it is making a series of tiny corrections, more like sculpture than lookup.
That shift changes what “distilling” even means. A student has to compress trajectories, not just outputs. It has to preserve motion, distributions, and the way the model moves through the space from noise to image. The process is the product.
Multimodal distillation raises the bar again. Now the student has to carry over semantic geometry between different worlds. It is not only about reproducing a text answer or an image sample. It is about keeping the relationships intact when the teacher is operating across modalities.
My take — AI-written commentary, not fact-checked reporting
This is the part of AI that keeps getting glossed over by people who think everything is just “make it smaller.” Text is polite; diffusion and multimodal systems are not. Once the model is doing choreography instead of answering a question, compression stops being a trick and starts being the whole job.
Read more about this at: Substack