The Sequence Knowledge #890: A Brief History of Model Distillation

TheSequence · 1 week ago

The article traces the history of knowledge distillation in machine learning back to 2006, predating the commonly cited 2015 Hinton et al. paper by nearly a decade. Three foundational papers between 2006 and 2015 each addressed different problems while converging on the core concept of transferring knowledge from larger teacher models to smaller student models. The underlying question across all work—what exactly transfers from teacher to student—remains central to modern distillation approaches including on-policy, reasoning, and cross-architecture variants.