Learning from human preferences
OpenAI Blog
Researchers at Anthropic and DeepMind have developed an algorithm that learns human preferences by comparing pairs of proposed behaviors rather than requiring explicit goal functions to be written. The system evaluates which of two actions a human prefers, using this feedback to infer the underlying objective. This approach reduces risks from misaligned proxy goals or incorrectly specified objectives in AI systems.
Why it matters
One step towards building safe AI systems is to remove the need for humans to write goal functions, since using a simple proxy for a complex goal, or getting the complex goal a bit wrong, can lead to undesirable and even dangerous behavior. In collaboration with DeepMind’s safety team, we’ve developed an algorithm which can infer what humans want by being told which of two proposed behaviors is better.
