Making LLMs faster without sacrificing accuracy

Amazon Science · 2 months ago

Researchers presented a framework at ICLR that extends Google DeepMind's Chinchilla scaling law to include architectural design choices for LLMs, enabling better speed-accuracy tradeoffs. Models with identical parameter counts can differ by up to 40% in inference throughput depending on hidden size, MLP-to-attention ratio, and grouped-query attention configuration. The resulting Surefire model family achieves 12-47% throughput improvements over LLaMA-3.2 while maintaining comparable accuracy by optimizing these architectural factors.