Can we change an AI model’s architecture without retraining it from scratch?
Amazon Science
Amazon says it can swap parts of a Transformer for cheaper layers without training from zero. That cuts pretraining cost by more than 99% and still keeps long-context reasoning intact.
Based on reporting by Amazon Science — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Amazon Web Services is making a pretty specific bet: you don’t have to rebuild a model’s brain from scratch just to change how it remembers things. In a new paper, the team proposes “priming,” a way to start hybrid models from pretrained Transformer weights instead of spending the usual mountain of tokens and GPU hours on every new architecture idea.
That matters because the search space for these hybrids has exploded. Some layers can keep full verbatim recall through attention, while others can get by with a fixed-size summary of the past. The source article’s point is simple: a lot of those local layers don’t need the expensive, ever-growing KV cache that comes with attention, so they can be replaced with state space layers that are lighter and faster.
Priming lets Amazon test those swaps cheaply. The paper says the method reuses the source Transformer’s learned projections, then does a short distillation step and a final end-to-end fine-tune. According to the article, the whole pipeline uses less than 0.5% of the source model’s pretraining token budget, and it works across Qwen, Llama, and Mistral families, including dense and MoE models. That is a very different kind of experiment from the usual “train it all again and hope” routine.
The results line up in a fairly sharp order. Among the three state space families tested, Gated KalmaNet comes out best, then Gated DeltaNet, then Mamba-2. The paper also says state space models beat sliding-window attention even when the sliding window gets two to four times more memory. On a 32-billion-parameter setup, a GKA-based hybrid matches its source Transformer on reasoning benchmarks while decoding up to twice as quickly with 128,000 tokens of context, and it reaches correct answers in as little as two-thirds of the wall-clock time.
Amazon has also released a zoo of primed hybrid models at the eight-billion and 32-billion scales, along with training code, optimized kernels, and a vLLM-serving plugin under Apache 2.0. That is the real tell here: this isn’t just a paper about architecture theory, it is an attempt to make architecture choice something people can actually iterate on without lighting a data center on fire.
My take — AI-written commentary, not fact-checked reporting
The industry loves pretending every architecture choice needs a heroic retrain. It usually doesn’t; it needs less mythology and more engineering. Amazon’s move is useful not because it invents a magic model, but because it makes the expensive ritual of “start over” look increasingly lazy.
Read more about this at: Amazon Science