Transformer²: Self-Adaptive LLMs
Sakana AI
Sakana AI built an LLM that adjusts its own weights on the fly per task, no retraining needed. It beats LoRA using way fewer extra parameters, and skills can even transfer between different base models.
Based on reporting by Sakana AI — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Sakana AI just published research on something they call Transformer² — yes, with the little superscript two — and the pitch is bold: an LLM that reshapes parts of itself depending on what you throw at it, in real time, without a retraining cycle in sight. Think less "fine-tune once and freeze" and more "adjust on the fly, task by task."
The trick sits in the math underneath. Sakana's team uses singular value decomposition to break a model's weight matrices into independent components — like isolating the neural pathways a brain might use for arithmetic versus language versus visual reasoning. Then, using reinforcement learning, they train compact "z-vectors" that dial each component up or down depending on the job at hand. A z-vector for GSM8K math might crank up one component and mute another; a z-vector for reasoning tasks flips a totally different combination. At inference time, the system runs a quick first pass to figure out what kind of task it's looking at — via a prompt classifier, a trained detector, or few-shot weight tuning — then applies the right blend of z-vectors before generating its actual answer.
The numbers are the interesting part. Tested on Llama and Mistral models across math (GSM8K, MATH), coding (MBPP-Pro, HumanEval), reasoning (ARC-Easy/Challenge) and visual QA (TextVQA, OKVQA), Transformer² beat LoRA, and did it while adding far fewer extra parameters. GSM8K saw especially strong gains, which the team credits to RL training not requiring a
My take — AI-written commentary, not fact-checked reporting
I like that this attacks a real bottleneck — static fine-tuning is expensive and brittle — instead of just scaling params again, and the cross-model z-vector transfer between Llama and Mistral is the detail everyone should be watching, because if task-specific skills really can hop between architectures, that's a much bigger deal than another benchmark win, though I'd want independent replication before buying the "living intelligence" framing wholesale.
Read more about this at: Sakana AI