PipelineRL

Hugging Face Blog · 1 year ago

ServiceNow Research open-sourced PipelineRL, a reinforcement learning system for large language models that updates model weights during inference without stopping the process, solving the traditional trade-off between inference speed and on-policy data collection. The system matched or exceeded Open-Reasoner's performance on AIME 2024 and MATH 500 benchmarks while using a simpler algorithm, training a 7B model in 3.5 days on 2 nodes and a 32B model in 6 days on 4 nodes. The modular architecture allows researchers to swap different inference and training software components as they improve.