Train separately, merge together: Modular post-training with mixture-of-experts

Allen Institute (AI2) · 2 months ago

Researchers from AI2 introduced BAR (Branch-Adapt-Route), a method for modular post-training that trains separate domain experts through independent pipelines and merges them using mixture-of-experts architecture. The approach achieved an average score of 49.1 across 19 benchmarks, outperforming monolithic post-training-only baselines (47.8) while enabling individual experts to be upgraded independently without retraining the entire model. This enables linear cost scaling for domain updates compared to quadratic costs in traditional monolithic retraining, allowing teams to upgrade specific capabilities like code or math without affecting others.