The Sequence Knowlege #907: The Brain Transplant: Distilling Transformers Into Other Architectures

Distillation used to be a pretty polite affair. A big transformer taught a smaller transformer, and everyone stayed in the family. Same attention layers, same basic wiring, just a difference in scale. The student got smaller or smarter or better at showing its reasoning, but it never stopped being a transformer underneath the hood.

Cross-architecture distillation throws that comfort out the window. Now the teacher is still a transformer, but the student is something else entirely — a state-space model, a linear RNN, some gated recurrent contraption that has never once computed an attention matrix. Researchers take a fully trained transformer, extract what it knows, and pour that capability into hardware that works on completely different principles. And the strange part is that it holds together. The knowledge survives the transplant, even though the receiving architecture processes sequences in a fundamentally different way, without the quadratic attention mechanism that made the teacher powerful in the first place.

The reason anyone bothers with this at all comes down to economics. Transformers are expensive at inference time because attention scales quadratically with sequence length, which becomes brutal once contexts stretch into the tens of thousands of tokens. State-space models and linear RNNs process sequences in something closer to linear time, which means dramatically cheaper and faster inference, especially on long documents or long conversations. If you could get transformer-level competence out of a linear-time architecture, you would have the intelligence without the electricity bill.

That is the arbitrage. Training a state-space model from scratch to match a top transformer is enormously expensive and often doesn't reach parity anyway, because the training recipes and data pipelines for these newer architectures are far less mature. But teaching an already-trained transformer's behavior to a fresh recurrent model turns out to be a shortcut nobody expected to work this well. The student never has to independently discover what good reasoning or good language modeling looks like — it just has to mimic outputs and internal signals from a teacher that already figured it out, using a body plan the teacher never had.

What makes this genuinely odd, rather than just clever, is that nobody fully understands why the transplant takes. Attention and recurrence are different computational strategies for handling context, not just different sizes of the same strategy. Yet capability seems to be somewhat separable from the specific mechanism that produced it, at least well enough for imitation to transfer real competence. That's an uncomfortable, interesting idea, and it's reshaping how people think about what these models are actually learning underneath all the architecture-specific plumbing.