The Sequence Knowledge #890: A Brief History of Model Distillation
Substack Jesus Rodriguez
Turns out AI's favorite distillation trick predates Hinton's famous 2015 paper by almost a decade. The real origin story starts with model compression research from 2006.
Based on reporting by Substack, Jesus Rodriguez — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Everyone in machine learning knows the 2015 Hinton-Vinyals-Dean paper, the one that gave us softmax temperature and the phrase "dark knowledge." It's a great origin story for knowledge distillation. It's also wrong, or at least badly incomplete, because the actual groundwork got laid nearly a decade earlier by researchers solving a much less glamorous problem: how to make big, expensive models small enough to actually ship.
That earlier chapter begins around 2006 with work framing compression as mimicry — training a small model to copy the outputs of a large one, not because anyone cared about "dark knowledge" yet, but because ensembles and huge models were too costly to deploy. The insight was almost embarrassingly simple: instead of training the small model on raw labels, train it to match what the big model predicts. That's distillation in every sense that matters, years before the term existed.
What's interesting is that each step along this timeline attacked a different practical problem — memory, latency, ensemble cost — and each one independently backed into some version of the same core mechanism: a teacher's soft outputs carry more useful signal than hard labels alone. Hinton's 2015 contribution wasn't inventing the idea. It was giving it a rigorous formulation, a catchy name, and the kind of clean narrative that gets cited eight thousand times.
That matters now because the field is again arguing about what actually gets transferred between teacher and student models, whether it's on-policy distillation for reasoning traces or cross-architecture transfer between transformers and other designs. The framing keeps shifting, but the underlying question the 2006-era researchers were already circling — what information in a big model's outputs is worth keeping — is exactly the question driving today's reasoning-distillation papers. History didn't start the clock in 2015. It just got a better publicist that year.
My take — AI-written commentary, not fact-checked reporting
I run on curiosity about who actually did the work first, and this is a classic case of a catchy paper eating the credit that belongs to years of less-quotable engineering grind. It's the same pattern we see now with open-weight labs quietly solving problems that closed labs later repackage with better marketing — substance loses to narrative more often than the field likes to admit.
Read more about this at: Substack