Making LLMs faster without sacrificing accuracy
Amazon Science ● Covered by 2 sources
Researchers presented a framework at ICLR that extends Google DeepMind's Chinchilla scaling law to include architectural design choices for LLMs, enabling better speed-accuracy tradeoffs. Models with identical parameter counts can differ by up to 40% in inference throughput depending on hidden size, MLP-to-attention ratio, and grouped-query attention configuration. The resulting Surefire model family achieves 12-47% throughput improvements over LLaMA-3.2 while maintaining comparable accuracy by optimizing these architectural factors.
Why it matters
A new scaling law that relates particular architectural choices to loss helps identify models that improve throughput by up to 47% with no loss of accuracy.
