Train separately, merge together: Modular post-training with mixture-of-experts
Allen Institute (AI2)
Researchers from AI2 introduced BAR (Branch-Adapt-Route), a method for modular post-training that trains separate domain experts through independent pipelines and merges them using mixture-of-experts architecture. The approach achieved an average score of 49.1 across 19 benchmarks, outperforming monolithic post-training-only baselines (47.8) while enabling individual experts to be upgraded independently without retraining the entire model. This enables linear cost scaling for domain updates compared to quadratic costs in traditional monolithic retraining, allowing teams to upgrade specific capabilities like code or math without affecting others.
Why it matters
BAR is a recipe for post-training language models one capability at a time—train domain experts independently, merge them into a single mixture-of-experts model, and upgrade any expert without impacting the others.
