Can LLMs invent better ways to train LLMs?
Sakana AI
Sakana AI got GPT-4 to invent a brand-new algorithm for training LLMs to follow human preferences, called DiscoPOP. An AI just designed a piece of AI training tech that beats human-made methods on real benchmarks.
Based on reporting by Sakana AI — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Researchers usually spend years tweaking the loss functions that teach language models to prefer one answer over another. Sakana AI decided to skip the years and ask an LLM to do the tweaking instead. In a new report done with the University of Oxford and Cambridge University, the company describes a pipeline where GPT-4 proposes a hypothesis, names a method, writes the code, watches how it performs, then tries again — over roughly 100 generations.
The standout result is something Sakana calls DiscoPOP, short for Discovered Preference Optimization. It emerged from a search that started on MT-Bench and was then checked against held-out tasks including AlpacaEval 2.0, so the team could confirm it wasn't just overfitting to one test. Across multiple benchmarks and base models, DiscoPOP outperformed Direct Preference Optimization and other existing hand-built methods, delivering higher reward scores while drifting less from the base model, measured by KL divergence.
What's odd is how the model got there. Sakana's team watched the LLM behave less like a random search engine and more like a working scientist: for one classification task it started with a label-smoothed cross-entropy objective, tuned the temperature, tried a squared-error variant that improved results, then fused the two ideas together for a bigger jump in performance. The written-out reasoning behind each step, and even the names the LLM gave its own inventions, tracked suspiciously well with what the code actually did. And DiscoPOP itself turned out to have an unconventional shape — it's not convex, which isn't how humans typically design these loss functions.
Sakana tested the code-generation step across GPT-4, Gemini-1.5, and Llama-3, and says all three could produce workable objective functions, though GPT-4 was the one used to establish what's currently possible. The team is now running similar experiments with open-source models and reports promising early results, with a stated goal of eventually closing the loop entirely on open models. Everything from checkpoints to the discovered loss functions to the discovery codebase itself has been published on GitHub and HuggingFace, so outside teams can poke at DiscoPOP or try running the discovery process themselves.
Sakana frames this as one piece of a bigger idea, sometimes called LLM², where models don't just get trained better but start contributing to the design of their own training methods, and eventually maybe their own architectures. That's a long way off. But a non-convex, LLM-authored loss function quietly beating DPO on held-out benchmarks is a concrete data point, not a thought experiment.
My take — AI-written commentary, not fact-checked reporting
An AI writing a better recipe for training AI sounds like science fiction until it's just a GitHub repo and a benchmark table, which is exactly what happened here. The part worth watching isn't DiscoPOP itself but Sakana's admission that open-source models are already showing promising results in the same pipeline — if that holds up, the interesting research stops being locked behind whichever lab has the biggest frontier model. Convexity was never sacred; if a weird, non-convex loss function trains better models with less drift, engineers should get over their attachment to tidy math.
Read more about this at: Sakana AI