Bridging Spherical Black-Box Optimizers

Sakana AI

Sakana AI researchers demonstrated that parametric and nonparametric black-box optimization methods share the same underlying mathematical framework, enabling hybrid optimizers for tasks like foundation model merging. The team developed two hybrid optimizers, AdaPol and SchedPol, that reduced computational costs for large language model merging by finding multiple solutions on smaller evaluation datasets instead of overfitting with standard methods. This theoretical unification allows engineers to design custom optimizers tailored to specific tasks while reducing the computational overhead of evaluating large models.