Data-Efficient Adaptation of LLMs via Attention Head Reweighting

arXiv cs.AI · 18 hours ago

Researchers proposed Attention Head Reweighting (AHR), a method that adapts large language models to new text-classification tasks by learning a single scalar weight per attention head rather than modifying many parameters. AHR outperformed LoRA on limited-sample tasks while modifying only 0.0001% of model parameters, achieving 200-1000x parameter reduction. This approach enables more efficient model adaptation in data-scarce domains like security while providing interpretability into how attention heads contribute to few-shot learning.