Fine-tuning GPT-2 from human preferences

OpenAI Blog · 6 years ago

Researchers fine-tuned GPT-2 using human feedback across multiple tasks, with the model successfully adopting the preferences expressed by human labelers even when those preferences diverged from the researchers' own expectations. Summarization required 60,000 human labels while simpler text continuation tasks needed only 5,000 labels to align model behavior with labeler preferences. The work aims to develop safety techniques that prioritize extracting human values through direct human-machine communication rather than relying on pre-defined objectives.