Transformer²: Self-Adaptive LLMs

Sakana AI · 35

Researchers introduced Transformer², a machine learning system that dynamically adjusts its weights for different tasks using Singular Value Decomposition and reinforcement learning. The method learns task-specific z-vectors that modulate weight matrix components, requiring far fewer parameters than LoRA while achieving comparable or better performance on math, coding, reasoning, and visual tasks. This approach enables LLMs to adapt to new tasks at inference time without retraining, and z-vectors learned on one model can partially transfer to another model.