Bridging Spherical Black-Box Optimizers

Black-box optimization has always felt like two separate tribes stuck in the same room. On one side, evolution strategies and their parametric cousins, which scale beautifully to huge search spaces but stubbornly converge on a single answer. On the other, consensus-based methods that happily juggle multiple solutions at once but choke once the dimensionality climbs past a few hundred parameters. Sakana AI's new ICML 2026 paper argues that this split was never as fundamental as it looked.

The team's core claim is that both families are really just different settings of one underlying update rule. Once you see ES and CBO as points on the same spectrum rather than separate species, you can start dialing between them, building hybrid optimizers tuned for whatever tradeoff a problem demands. That's not just a tidy theoretical unification; it's a design space Sakana says it can now exploit directly.

The practical test case is merging large language models, an area Sakana has been chipping away at since its earlier evolutionary model-merging work. The catch with LLM merging is brutally simple: evaluating a candidate merge properly means running it against a big dataset, which is slow and expensive. Shrink the dataset to save compute, and standard single-solution optimizers start overfitting to that smaller sample instead of finding merges that actually generalize.

Sakana's fix was to stop treating merging as a search for one best answer and instead frame it as a multimodal problem, one with many good solutions scattered across the landscape of possible merges. Using two new hybrid optimizers born from their unified framework, called AdaPol and SchedPol, they found several distinct high-quality optima on the cheaper, smaller dataset rather than latching onto a single overfit peak. The result: merges that hold up on the full evaluation, achieved without paying for constant large-scale testing along the way.

It's a modest-sounding trick with a fairly big implication. If overfitting on small evaluation sets has been quietly limiting how cheaply people can merge or tune models, a multimodal search strategy borrowed from decades-old optimization theory could shave real compute costs off a process that's becoming central to how foundation models get built.