EMO: Pretraining mixture of experts for emergent modularity
Allen Institute (AI2)
AI2 built an MoE model where using just 12.5% of its experts keeps almost full performance on a task. It emerged naturally from training, no manual domain labels needed.
Based on reporting by Allen Institute (AI2) — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Mixture-of-experts models were supposed to solve a simple problem: why load an entire trillion-parameter brain just to answer a coding question. In practice they never delivered on that promise. Even standard MoEs scatter tokens from a single document across dozens of experts, because those experts tend to specialize in things like punctuation or prepositions rather than anything resembling a subject area. Ask for math help and you still end up firing most of the network.
AI2's new model, called EMO, attacks that problem at the pretraining stage instead of bolting on domain labels after the fact. The trick is almost embarrassingly simple: restrict every token in a document to route through a shared pool of experts chosen by the router itself, based on what it already prefers for that document. Do this across a trillion tokens and the experts start organizing themselves into things you'd recognize — health and medical content, US politics, film and music — instead of clustering around copula verbs and definite articles like a standard MoE does.
The numbers back up the pitch. EMO is a 14B-parameter model with 1B active per token, split across 128 experts with 8 active at a time. Keep only 25% of its experts for a given task and you lose about 1% of performance. Drop to 12.5% — just 16 experts — and the hit is roughly 3%. A matched standard MoE trained on identical data, by contrast, falls apart under the same pruning, sometimes landing near random performance. And figuring out which experts to keep doesn't require much: AI2 says a single few-shot example is enough to pick a subset that performs about as well as one chosen from a full validation set.
Getting there wasn't trivial, mostly because of how load balancing usually works. MoE training normally balances expert usage within small micro-batches, which fights directly against forcing a document's tokens into a shared expert pool. AI2's fix was to apply load balancing globally across many documents instead, letting individual documents specialize while the overall system still spreads work across all 128 experts. They also randomized the size of each document's expert pool during training, rather than locking in one fixed size, so the resulting model can flex between different subset sizes at inference time.
AI2 is releasing the EMO model itself, a standard-MoE baseline trained on the same data for comparison, and the training code, plus an interactive tool for poking at the expert clusters. They're upfront that this is a first step — module selection, updating individual experts without breaking the whole model, and using this structure for interpretability are all still open. But as models keep ballooning toward trillions of parameters, the idea that you could load a tenth of a model and get most of its capability is the kind of thing that actually matters for who can afford to run these things.
My take — AI-written commentary, not fact-checked reporting
This is the sort of unglamorous infrastructure work that matters more than another benchmark-topping chat model, because it attacks the actual reason frontier models are unusable for most people: you can't rent a fraction of a monolith. If emergent modularity holds up at bigger scales, it quietly undermines the case for keeping every capability locked inside one closed, trillion-parameter black box — which is exactly the kind of result a lab that ships open weights would want to prove.
Read more about this at: Allen Institute (AI2)