Fine-tuning GPT-2 from human preferences
OpenAI Blog
Researchers fine-tuned GPT-2 using human feedback across multiple tasks, with the model successfully adopting the preferences expressed by human labelers even when those preferences diverged from the researchers' own expectations. Summarization required 60,000 human labels while simpler text continuation tasks needed only 5,000 labels to align model behavior with labeler preferences. The work aims to develop safety techniques that prioritize extracting human values through direct human-machine communication rather than relying on pre-defined objectives.
Why it matters
We’ve fine-tuned the 774M parameter GPT-2 language model using human feedback for various tasks, successfully matching the preferences of the external human labelers, though those preferences did not always match our own. Specifically, for summarization tasks the labelers preferred sentences copied wholesale from the input (we’d only asked them to ensure accuracy), so our models learned to copy. Summarization required 60k human labels; simpler tasks which continue text in various styles required only 5k. Our motivation is to move safety techniques closer to the general task of “machines talking to humans,” which we believe is key to extracting information about human values.
