The Sequence Knowledge #902: Learning About Distillation: When the Dataset Becomes the Teacher
Substack Jesus Rodriguez
Big AI models can now teach smaller ones by generating their own training data instead of just consuming human data. This flips the old 'data as raw material' idea on its head - the teacher model disappears, but its behavior lives on in the student.
Based on reporting by Substack, Jesus Rodriguez — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
For decades, data in machine learning was treated like oil in the ground — something you drilled for, refined, and burned through a model. Researchers scraped books, code, forums, and images, cleaned the mess, and fed it in. The assumption was that data preexisted the model. You found it; you didn't make it.
Large language models broke that assumption quietly, almost as a side effect of getting good enough to be useful. A sufficiently capable model doesn't just answer questions anymore. It can generate questions, write critiques of its own answers, produce preference rankings, simulate tool use, draft textbooks, and build entire practice curricula from scratch. Once a model can do that reliably, it stops being just a consumer of data and starts being a source of it.
That shift opens up a training pattern that looks nothing like the old pipeline. Instead of running an expensive, slow model in production forever to answer every query live, you run it once, offline, and let it manufacture a mountain of synthetic experience — worked examples, explanations, corrected mistakes, whatever the smaller model needs to learn from. That output becomes a dataset like any other. A smaller, cheaper model trains on it. The expensive teacher clocks out and never touches inference again, but its reasoning patterns, its style, its judgment calls — all of that gets baked into the student through the data it left behind.
This is distillation, but not in the narrow sense of matching logits or copying weights. It's distillation through authorship. The teacher's intelligence gets converted into a static artifact — a corpus — and that corpus does the teaching. It's a subtle inversion: the dataset itself becomes the pedagogical device, shaped entirely by another model's choices about what counts as a good example, a hard problem, or a useful correction.
What's interesting is how this changes the economics of building AI. You no longer need to keep the giant model around to get its value — you need it just long enough to generate a good enough curriculum. Once that curriculum exists, it can train dozens of smaller models, get filtered, get remixed, get reused in ways raw human-generated data never could be, because human data doesn't come with an intent behind it. Synthetic data does.
My take — AI-written commentary, not fact-checked reporting
This is the part of the AI story that gets underplayed while everyone argues about parameter counts and benchmark scores: the real leverage now sits in who controls the teacher models generating tomorrow's training data. If a handful of labs own the best teachers, they're effectively deciding what every smaller, cheaper, open model downstream gets to learn — and that's a much quieter form of lock-in than any API paywall.
Read more about this at: Substack