The N Implementation Details of RLHF with PPO
Hugging Face Blog
Researchers reproduced OpenAI's 2019 RLHF implementation with PPO by studying the original codebase and documented the engineering details needed to match its results on stylistic tasks. Their reproduction codebase produces learning curves nearly identical to OpenAI's original code when training on sentiment and descriptiveness tasks. The work provides a checklist of implementation details and a minimal reference implementation to help others understand RLHF engineering without requiring the specific hardware (8 V100 32GB GPUs) that OpenAI's original code demands.
