TAID: A Novel Method for Efficient Knowledge Transfer from Large Language Models to Small Language Models

Sakana AI's newest research paper landed a Spotlight slot at ICLR 2025, one of machine learning's biggest annual gatherings, for a technique called TAID — Temporally Adaptive Interpolated Distillation. The pitch is straightforward: take a giant language model's knowledge and pour it into a much smaller one, without losing what made the giant good in the first place. The proof-of-concept is TinySwallow-1.5B, a Japanese small language model built with the Swallow team at the Institute of Science Tokyo, which the company says now achieves state-of-the-art performance among similarly sized models.

Traditional knowledge distillation works by having a smaller "student" model imitate a larger "teacher" model — not just copying final answers but absorbing the teacher's probability distributions over choices. Sakana's example: asked to complete "Sakana AI develops efficient methods for ___," a teacher model might assign 35% probability to "AI," 25% to "ML," and 15% to "LLM," and passing along that full spread teaches the student more than a single correct label ever could. The catch, well documented in prior research, is that bigger teachers aren't automatically better teachers. When the gap between teacher and student capacity gets too wide, transfer breaks down — like handing graduate coursework to a grade-schooler.

TAID's fix is a moving intermediate teacher that sits between student and teacher, initially close to what the student can handle and gradually shifting toward the full teacher model as training proceeds. In tests with 70-million-parameter student models pretrained on a 1-billion-token slice of the SmolLM-Corpus and evaluated on the LAMBADA benchmark via lm-eval-harness, conventional distillation methods (KL and RKL) actually got worse as teacher models grew larger. TAID kept improving instead, scaling with teacher size rather than fighting it.

The real test came with TinySwallow-1.5B, distilled down from a 32-billion-parameter teacher into a 1.5-billion-parameter model — roughly a twentieth of the original size. Despite that shrinkage, Sakana says it runs efficiently enough to generate text at solid speed directly on an iPhone 14, and the company built a browser-based Japanese interface that runs the model client-side in JavaScript, no external API required. An English counterpart, TAID-LLM-1.5B, exists too, and the underlying method has already been extended to vision-language models with TAID-VLM-2B, described as outperforming existing distillation approaches in that domain.

The work was done with Kyoto University's Han Bao and Sho Yokoi of the National Institute for Japanese Language and Linguistics, Tohoku University and RIKEN, and was supported through Japan's GENIAC program run by METI and NEDO. The paper is on arXiv, and the weights for TinySwallow-1.5B are on Hugging Face — Sakana is framing this less as a one-off product and more as evidence that small, efficient models don't have to mean compromised ones.