GSPO: Towards Scalable Reinforcement Learning for Language Models
Qwen
Researchers propose GSPO, a new reinforcement learning algorithm designed to address training instability issues in existing methods like GRPO that cause model collapse during extended training. The algorithm aims to maintain stable training dynamics while scaling language models, addressing a key bottleneck in improving performance with increased computational resources. GSPO enables more reliable long-term training of language models with reinforcement learning without the irreversible collapse seen in previous approaches.
Why it matters
PAPER DISCORD Introduction Reinforcement Learning (RL) has emerged as a pivotal paradigm for scaling language models and enhancing their deep reasoning and problem-solving capabilities. To scale RL, the foremost prerequisite is maintaining stable and robust training dynamics. However, we observe that existing RL algorithms (such as GRPO) exhibit severe instability issues during long training and lead to irreversible model collapse, hindering further performance improvements with increased compute. To enable successful RL scaling, we propose the Group Sequence Policy Optimization (GSPO) algorithm.
