Learning from human preferences

Writing a reward function sounds simple until you actually try it. Tell an AI to maximize some proxy score and it will often find the laziest, weirdest way to hit that number, technically satisfying you while doing the opposite of what you meant. That gap between what we say and what we mean is exactly where dangerous behavior sneaks in, and it's the problem OpenAI and DeepMind's safety team decided to attack together.

Their answer skips the reward-function-writing step entirely. Instead of a human sitting down and encoding a goal in math, the system shows a person two short clips of an agent behaving in some environment and simply asks which one looks better. No scoring, no formulas, just a preference. Do that enough times across enough comparisons, and the algorithm starts building an internal model of what 'good' looks like, then uses that model to train the agent going forward.

What makes this notable isn't the cleverness of the trick alone, it's what it implies for the alignment problem. If an AI can pick up complex, hard-to-specify goals from a trickle of human judgments rather than a hand-tuned proxy, you close off one of the more obvious paths to it optimizing for the wrong thing. The people writing the reward function are usually the weak link, not the AI itself, and this approach quietly removes them from that role.

The two labs frame it as a small but concrete step, not a solved problem. Preference-based learning still depends on humans giving consistent, meaningful feedback, and scaling that feedback to more complicated real-world tasks is its own open question. But the direction is clear enough: teach machines what we want by showing them examples of it, rather than trying to write down the perfect equation for wanting.