Evolved Policy Gradients
OpenAI
OpenAI built an AI that learns how to learn better by evolving its own reward math. The trained agents handled totally new setups they'd never practiced.
Based on reporting by OpenAI — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
OpenAI has published research on Evolved Policy Gradients, an experimental take on metalearning that skips the usual approach of tweaking a neural network's weights and instead evolves the loss function the agent uses to learn in the first place. It's a subtle but important distinction. Most reinforcement learning systems are handed a fixed reward signal and told to optimize against it. EPG instead treats the loss function itself as something that can be shaped by evolution, using it almost like DNA that gets passed down and mutated across generations of training runs.
The payoff shows up in generalization. Agents trained with EPG were tested on tasks that diverged from anything they saw during training, like a navigation problem where the target object sat somewhere in the room it had never been placed before. Traditional policy gradient methods tend to choke on this kind of shift because they've essentially memorized a narrow slice of the environment. EPG-trained agents adapted faster because the loss function guiding them had itself been optimized to handle a spread of situations, not just the ones baked into the original training data.
What makes this interesting is the two-loop structure underneath it. An outer evolutionary loop searches over possible loss functions, while an inner loop actually trains the agent using whichever loss function the outer loop is currently testing. Over many iterations, the outer loop favors loss functions that produce agents capable of learning quickly and robustly, effectively discovering a better recipe for learning rather than a better answer to a specific problem.
OpenAI is framing this as early-stage and experimental, not something ready for production robotics or game-playing systems tomorrow. But the direction matters. If loss functions can be evolved rather than hand-designed, that opens a path toward agents that adapt on the fly to environments their creators never explicitly anticipated, which has been one of the persistent weak spots in reinforcement learning research for years.
My take — AI-written commentary, not fact-checked reporting
I like this because it attacks the actual bottleneck in RL, which isn't compute or data, it's that we've been hand-crafting reward functions like it's still the 1980s. Evolving the loss function itself feels like the right level of abstraction to attack, even if EPG today is a toy compared to what it implies for five years out. My only gripe is OpenAI keeps publishing these clever primitives without much urgency toward real-world deployment, which is fine for research but let's not pretend it's close to robots navigating your actual living room.
Read more about this at: OpenAI