Making LLMs faster without sacrificing accuracy
Amazon Science ● Covered by 2 sources
Amazon researchers built a scaling law that picks LLM architecture, not just size, to boost speed without hurting accuracy. Their Surefire models match LLaMA-3.2 accuracy while running up to 47% faster.
Based on reporting by Amazon Science — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Everyone obsesses over parameter counts and training tokens when talking about LLM scaling, but Amazon's ICLR paper makes a case that we've been ignoring half the equation. Two models with identical parameter counts, trained on identical data, reaching identical loss, can still differ by 40% in inference throughput. That gap comes down to architecture — hidden size, how parameters get split between attention and MLP layers, and whether you use grouped-query attention — choices the famous Chinchilla scaling law from DeepMind never addressed.
The Amazon team extended Chinchilla's framework by adding these architectural knobs as first-class variables rather than afterthoughts. They found that hidden size and the MLP-to-attention ratio both follow U-shaped loss curves, meaning there's a sweet spot for each, and that these two factors can be tuned independently because their effects on loss don't interact. Grouped-query attention behaves less predictably, so they handled it separately through local search rather than a clean formula.
To prove it out, the team trained more than 200 models ranging from 80 million to 3 billion parameters and produced two resulting families: Panda, tuned purely for accuracy, and Surefire, tuned for the best accuracy-per-throughput tradeoff. The numbers are striking. Surefire-1B roughly matches LLaMA-3.2-1B's accuracy while pushing throughput up 21% on vLLM and 47% on SGLang. Surefire-3B does the same at the 3-billion scale, gaining 12-17% throughput with essentially no accuracy loss. Panda models, meanwhile, squeeze out small accuracy wins — 2.1% at 1B, 0.6% at 3B — at the cost of slower inference, showing the tradeoff runs both directions.
Maybe the most useful finding, though, is a practical one: this scaling law works even when calibrated on tiny models, some with as few as 80 million parameters, and still correctly predicts the best architecture at 1B and 3B scale. That means labs can run cheap experiments to find good architectures before committing to expensive full-size training runs. And the gains held up consistently across A100 and H200 GPUs, and across vLLM and SGLang serving stacks, which is exactly the kind of robustness that turns a research paper into something engineering teams actually adopt.
The underlying message is blunt: existing open-weight models like LLaMA-3.2 are over-invested in MLP layers relative to attention, with ratios near 4.8 where the math suggests something closer to 1.0 is optimal. That's not a minor inefficiency — it's free throughput sitting on the table for anyone willing to rethink architecture instead of just scaling up parameters and tokens.
My take — AI-written commentary, not fact-checked reporting
I like papers like this because they quietly undercut the 'just add more parameters' orthodoxy that's dominated LLM development since GPT-3. Amazon isn't claiming a breakthrough model, they're showing that most labs have been leaving 20-40% throughput on the table through sloppy architectural defaults, which is a much bigger deal for anyone actually running inference at scale than another leaderboard win. If this holds up outside Amazon's own benchmarks, expect the open-weight crowd to start treating MLP-to-attention ratio as seriously as they treat token counts.
Read more about this at: Amazon Science