Population-based Model Merging via Quality Diversity

Sakana AI has a new paper out, and it argues for something almost contrarian in an industry obsessed with scale: instead of building one enormous model that tries to do everything, why not evolve a population of small, specialized ones that divide the work up like species filling different ecological niches. The framework is called CycleQD, and it comes out of research backed by Japan's Ministry of Economy, Trade and Industry through its GENIAC supercomputing grant.

The method borrows from Quality Diversity, an evolutionary computing approach that cares less about finding the single best solution and more about building a varied collection of excellent solutions, each strong in its own way. Sakana's team applies this to language models by starting with expert LLMs, each specialized in one skill, and then merging pairs of them together as a kind of crossover, followed by a mutation step using Singular Value Decomposition to nudge the new model into fresh territory the parent models hadn't explored. The twist that gives CycleQD its name is that the target skill being optimized rotates: one skill becomes the quality metric while the others serve as behavior characteristics, then the focus cycles to the next skill, and so on.

Sakana tested this by training Llama3-8B-Instruct on three computer science tasks: coding using the MBPP benchmark, database operations, and operating system operations. CycleQD beat both traditional fine-tuning and Sakana's earlier model-merging techniques across these tasks, according to results the company laid out in a comparison table alongside GPT models included for reference. The team also visualized the resulting population, plotting models by their performance across these skill dimensions, and found that the final population wasn't just a scattering of narrow specialists. Many of the evolved models turned out capable across multiple tasks at once, filling out the grid far beyond the three starting points marked in the visualization.

What's notable is that Sakana frames this as more than a coding benchmark trick. The company says the approach extends into non-language domains too, pointing to image segmentation as an example, and pitches the whole framework as a more sustainable alternative to the compute-hungry race toward ever-larger foundation models. Looking ahead, Sakana ties CycleQD to two bigger ambitions: continual, life-long learning systems that keep accumulating skills over time, and multi-agent setups where a diverse swarm of specialized agents could collaborate or compete on problems no single generalist model handles well.