TAID: A Novel Method for Efficient Knowledge Transfer from Large Language Models to Small Language Models

Sakana AI

Sakana AI introduced TAID, a knowledge distillation method that transfers knowledge from large language models to smaller ones by adapting the teacher model based on student progress. The method was validated by creating TinySwallow-1.5B, a Japanese language model compressed from 32 billion to 1.5 billion parameters while achieving state-of-the-art performance for its size. TAID enables compact models to run on edge devices like smartphones, making AI more accessible without requiring massive computational resources.