The Sequence Knowledge #886: Demystifying Model Distillation

There's a teacher and there's a student, and the whole business of model distillation comes down to that relationship. The teacher is the big, expensive, slow model — think GPT-4 class or bigger — full of capacity but a pain to run at scale. The student is the small model everyone actually wants to deploy: cheap, fast, easy to stick on a phone or a server rack without melting the budget. Left alone, that student trained on raw data usually ends up mediocre. Distillation exists to fix that gap.

The trick, as this piece frames it, is deceptively simple. Instead of training the small model purely on the original dataset — raw text, raw labels, ground truth as humans wrote it — you train it on the big model's interpretation of that data. The teacher doesn't just spit out final answers; it produces soft probability distributions, confidence spreads across many possible outputs, and that richer signal turns out to carry far more information than a single correct label ever could. A student learning from those distributions is effectively learning how the teacher thinks, not just what it concludes.

This matters more now than it did a few years ago because everyone building production AI systems is chasing the same tradeoff: capability versus cost. Frontier labs don't want to serve their biggest model for every mundane query, so they distill it down into smaller variants that inherit a surprising amount of the original's judgment. That's a big part of why so-called small language models have gotten unnervingly competent over the past year or two — many of them were never trained from scratch in isolation, they were trained in the shadow of a much larger sibling.

What the piece is really doing, ahead of getting into the technical weeds, is reframing distillation away from the sci-fi framing of 'copying a brain' and toward something more mundane and more useful: a supervised learning setup where the labels happen to come from another neural network instead of a human annotator. Once you see it that way, the appeal is obvious. Human-labeled data is slow and expensive to produce; a well-trained teacher model can generate an effectively unlimited stream of nuanced supervision signal, at whatever scale the student needs to learn from.