AMD Releases Instella-MoE-16B-A3B: A Fully Open Mixture-of-Experts LLM With 2.8B Active Parameters Trained On Instinct GPUs
MarkTechPost Asif Razzaq ● Covered by 33 sources
AMD trained a full AI model start-to-finish on its own MI300X chips instead of Nvidia's, and it beats other open models on benchmarks. Catch: the license blocks commercial use for now.
Based on reporting by MarkTechPost, Asif Razzaq — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
AMD just gave the AI world a fairly rare thing: a large language model trained entirely from scratch on its own hardware, with the receipts to prove it. Instella-MoE-16B-A3B carries 16 billion total parameters but only lights up 2.8 billion of them for any given token, using a mixture-of-experts setup with two shared experts and six picked from a pool of 64. The whole pipeline, every checkpoint, every data mixture, every training config, ran on Instinct MI300X and MI325X accelerators, which is the part AMD really wants people to notice.
Two engineering tweaks do a lot of the heavy lifting here. Gated Multi-head Latent Attention bolts a small learned gate onto the attention output, letting the model dial signal up or down before it hits the final projection. FarSkip-Collective, meanwhile, is a scheduling trick that feeds slightly stale or partial activations into the MoE and attention layers so that communication between experts can overlap with computation instead of blocking it. AMD claims this combo delivers a 12.7% speedup in pre-training and cuts time-to-first-token by as much as 39.2% during expert-parallel serving. Those aren't huge headline numbers, but for anyone running MoE models at scale, shaving communication overhead is exactly where the money gets saved.
The training recipe itself is a marathon: 7.1 trillion tokens of pre-training pulled from open corpora like Nemotron-CC-v2 and MegaMath, a mid-training phase on Dolma3 Dolmino data, a context stretch from 4K up to 64K tokens using YaRN, and then a post-training gauntlet of supervised fine-tuning, DPO, and reinforcement learning through AMD's Miles framework. The RL stage is notable for stacking instruction-following gains on top of a distillation step designed specifically so the model doesn't forget math and code skills along the way — a failure mode that trips up a lot of RL-tuned models.
On the numbers, Instella-MoE holds its own. Its base checkpoint averages 76.7 across benchmarks, edging out Moonlight-16B-A3B and comfortably beating SmolLM3, OLMo-3, and OLMoE, though it still falls short of Qwen3.5-4B-Base's 79.5. The instruction-tuned
My take — AI-written commentary, not fact-checked reporting
: itself its progressed reasoning variant climbs from 71.58 after SFT to 73.22 after the full RL pipeline, ahead of comparable Olmo3 and Gemma models. Long-context handling at 64K also holds up reasonably well on HELMET and RULER. Here's the catch, and it's a real one: the weights ship under AMD's ResearchRAIL license, which is academic-and-research-only, not a license you can build a product on. The training code is MIT licensed and fully reusable, which is arguably the more valuable gift to the field — a working, documented recipe for training a competitive MoE model end to end on non-Nvidia silicon. But if you wanted to just download the weights and ship a commercial chatbot tomorrow, this isn't that.this this it isn't that release..hridge, product.. ounded'-thatchnical around.
Read more about this at: MarkTechPost