The Sequence Knowledge #898: The Trace Is the Teacher: Distilling Reasoning Into Small Models
Substack Jesus Rodriguez
DeepSeek fed 800K messy reasoning transcripts from its R1 model into small open models with plain fine-tuning, no RL involved. The tiny models started solving hard math and catching their own errors — something distillation theory said couldn't happen this way.
Based on reporting by Substack, Jesus Rodriguez — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Back in January 2025, DeepSeek did something almost embarrassingly simple with its big reasoning model, R1. They had it generate roughly 800,000 worked-out solutions, complete with dead ends, backtracking, and the occasional "wait, let me reconsider" moment. They kept the ones that were correct and readable, then ran plain old supervised fine-tuning on a lineup of small open models: Qwen at 1.5B, 7B, 14B, and 32B, plus Llama at 8B and 70B. No reinforcement learning. No reverse KL tricks. No teacher grading the student's own outputs. Just next-token prediction on somebody else's transcripts.
The results were not modest. The 32B model started cracking competition-level math problems that had no business being within reach of a model that size. The 7B model, on its own, began checking its work mid-answer and branching into alternate reasoning paths — behaviors nobody explicitly programmed in. A handful of unremarkable dense models suddenly punched well above their weight class, just from reading someone else's scratch paper.
Here's the awkward part. There's a whole body of distillation theory built around the idea that copying a teacher's exact trajectories is a trap: at inference time the student wanders off the teacher's path, hits states it's never seen, and the training signal falls apart. That's the entire justification for reverse KL and on-policy sampling in the first place. So watching the most consequential reasoning-distillation result in years turn out to be dead-simple sequence imitation feels like it should break the argument. Either imitation works fine, or it doesn't — and this looked like proof it does.
Except it's neither. What DeepSeek's traces actually contain isn't a fixed answer to memorize — it's a demonstration of the process of reasoning itself: how to notice a mistake, how to try a different angle, how to talk yourself through uncertainty. A student trained on that doesn't need to stay on the teacher's exact path, because what it's absorbing is a transferable skill for navigating its own paths, not a script to recite. The trace is the teacher, not the destination — and that distinction is why 800,000 messy transcripts turned six ordinary models into something that reasons far beyond their size.
My take — AI-written commentary, not fact-checked reporting
This is the kind of result that should make people rethink how obsessed the field got with elaborate RL pipelines for reasoning. Sometimes the unglamorous move — dump good raw thinking into a model and let it learn the shape of thought — beats the clever machinery, and it's a reminder that open-weight releases of things like R1's traces do more for the ecosystem than another closed API ever will.
Read more about this at: Substack