GRPO Beyond English: A Large-Scale Study of GRPO in Non-English and Multilingual Settings
Apple Machine Learning Research
Researchers conducted a large-scale study of Group Relative Policy Optimization (GRPO), a reinforcement learning method for improving language model reasoning, across multiple languages beyond English. Training models to reason in their native language closes the performance gap with English reasoning, and training in one language often improves performance in others through crosslingual transfer. However, improvements are model- and language-dependent, with some cases showing severe performance regressions in other languages, requiring broader evaluation to detect language-specific degradation.
Why it matters
Reinforcement Learning with Verifiable Rewards (RLVR), often optimized with Group Relative Policy Optimization (GRPO), has become a central recipe for improving the reasoning capabilities of pretrained language models but current studies remain heavily English-centric. We conduct a large-scale empirical study of multilingual and non-English GRPO across a wide range of base models, training languages, and different reasoning language rewards. We find that training to reason in the native language often leaves only a small gap to training for English reasoning. We further observe strong…