GSPO: Towards Scalable Reinforcement Learning for Language Models

GitHub Pages · 1 year ago · 36

Researchers propose GSPO, a new reinforcement learning algorithm designed to address training instability issues in existing methods like GRPO that cause model collapse during extended training. The algorithm aims to maintain stable training dynamics while scaling language models, addressing a key bottleneck in improving performance with increased computational resources. GSPO enables more reliable long-term training of language models with reinforcement learning without the irreversible collapse seen in previous approaches.