Gathering human feedback

Reward functions are the quiet tyrants of reinforcement learning. You write a formula, the agent optimizes it, and if you got the formula even slightly wrong, the agent finds the loophole with ruthless efficiency. OpenAI's answer, released as an open-source project called RL-Teacher, sidesteps the problem by cutting the formula out of the loop entirely and putting a human in it instead.

The idea is simple to describe even if it's fiddly to build: instead of hand-crafting a score for every action an agent takes, you show a person short clips of the agent's behavior and ask which one looks better. Those comparisons get folded into a model that predicts what a human would prefer, and that prediction becomes the reward signal driving the actual training. No human has to watch every frame or grade every action. Occasional spot checks are enough to steer the whole process.

This matters most for tasks where nobody can write down what 'good' means in code. Think of behaviors that are easy to recognize but painful to formalize, like a robot moving in a way that seems natural, or an agent avoiding actions that are technically legal but obviously undesirable. Hard-coded rewards tend to miss these nuances, and agents trained on them often exploit the gaps. Feedback from a person, even sparse feedback, captures judgment that's hard to encode any other way.

OpenAI frames this less as a productivity trick and more as a safety mechanism. If reinforcement learning systems keep expanding into messier, real-world domains, the ability to align an agent's behavior with what humans actually want, rather than with a proxy that only approximates it, becomes a core requirement rather than a nice-to-have. Releasing RL-Teacher as open source lets other researchers plug human feedback into their own environments instead of starting from a blank page.