The Sequence Knowledge #894: When the Student Started Talking Back: Distillation in the LLM Era
Substack Jesus Rodriguez
Old-school AI distillation just shrank a model by copying its answers. Now teacher models are teaching students to actually think, not just mimic.
Based on reporting by Substack, Jesus Rodriguez — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Back in 2015, distillation was almost tidy. A big teacher model spat out a probability vector, a small student model tried to match it, you ran backprop, and called it a day. The whole setup assumed a fixed set of inputs and a closed set of outputs. It worked because the world it operated in was small and well-behaved.
Large language models blew that up. There's no closed set of classes when your output space is every possible next token in every possible sentence. There's no single fixed input distribution when a model might be asked to write code, summarize a contract, or argue about ethics in the same afternoon. The neat 2015 pipeline assumed a kind of mechanical innocence that simply doesn't survive contact with sequence generation.
What TheSequence traces in this piece is the five-year unraveling of that innocence, broken into three stages. The first stage is the recognition that sequences aren't pictures — you can't just distill a classifier's output and expect that to transfer to something producing chains of tokens, where every wrong step compounds into the next. Matching a single probability vector stops making sense when the object you're trying to transfer is a reasoning process rather than a snapshot judgment.
And that's the real pivot the piece is after: distillation quietly stopped being about compression — building a smaller copy of a fixed function — and became about capability transfer. The goal isn't shrinking a model anymore. It's getting a smaller model to actually do the hard thing a bigger one can do, using the big model's outputs as a curriculum rather than a target to imitate pixel-for-pixel.
Each of these stages, per the analysis, looked like a small engineering tweak in the moment. Nobody was announcing a paradigm shift when they started training students on teacher-generated text instead of teacher-generated probabilities. But stacked together, these tweaks moved what distillation even means. The student stopped being a lossy copy and started being trained to reason, at least in miniature, the way the teacher does.
My take — AI-written commentary, not fact-checked reporting
I like framing this as capability transfer rather than compression, because that's the distinction most people miss when they wave away small models as 'just distilled.' A student model that's learned to reason through a teacher's chain of thought isn't a cheap knockoff — it's a genuinely different kind of artifact, and treating it like a compressed JPEG of GPT-4 undersells what's actually happening under the hood.
Read more about this at: Substack