Mixture-of-Agents Alignment: Harnessing the Collective Intelligence of Open-Source LLMs to Improve Post-Training

Together AI has a new trick for making small language models punch way above their weight, and it doesn't involve begging OpenAI for API access. The company's Mixture-of-Agents Alignment (MoAA), detailed in a paper accepted at ICML 2025, takes several open-source models, has them collaborate on answers, then distills that collective output into a single compact model through fine-tuning. The headline result: Llama-3.1-8B-Instruct jumps from 19.5 to 48.3 on Arena-Hard after this treatment, and Gemma-2-9B-it climbs from 42 to 55.6, landing it close to Llama-3.1-70B-Instruct despite being roughly a tenth the size.

The mechanics are a two-stage pipeline. First, a panel of proposer models — WizardLM-2-8x22b, Gemma-2-7b-it, Qwen-2-72B-Instruct, and Llama-3.1-70B-Instruct — each generate a response to a prompt, and an aggregator model, Qwen-1.5-110B-Instruct, synthesizes those into one polished answer. That becomes the supervised fine-tuning data. Second, the team runs Direct Preference Optimization, but instead of a dedicated reward model they use the Mixture-of-Agents setup itself to judge which of five sampled responses is best and worst, then trains on that preference signal. Both stages compound: SFT alone gives a solid bump, DPO on top pushes performance further on AlpacaEval 2 and Arena-Hard, though it costs a bit on MT-Bench compared to using GPT-4o as the judge.

What's notable isn't just the score gain, it's that this synthetic data reportedly beats data generated by GPT-4o itself for the same fine-tuning job, and does it 15% cheaper — $366 versus $429 for generating roughly 60,000 examples from the UltraFeedback dataset. Together also tested swapping in various reward models — ArmoRM, PairRM, Llama-3.1-70B, Gemma-2-27B, Qwen2-72B — and found their MoA-as-judge approach outperformed several of them, plus GPT-4o, on two of three benchmarks.

The most interesting wrinkle is what happens when you feed a model its own committee's output. Researchers tried a smaller three-model mixture — Gemma-2-9B-it, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.3 — with Gemma-2-9B-it acting as both a proposer and the aggregator. Fine-tuning Gemma on data produced by that mixture still improved the model over its own baseline, even though it was one of the models generating the training data in the first place. That's the seed of a self-improving loop: models bootstrapping better versions of themselves without needing a bigger, closed-source teacher standing over their shoulder.

Together released the paper, the SFT datasets, and four model checkpoints — SFT and DPO versions for both Llama-3.1-8B-Instruct and Gemma-2-9B-it — so anyone can poke at the claims directly rather than take the benchmark charts on faith.