Improving mathematical reasoning with process supervision
OpenAI Blog
OpenAI trained a model using process supervision, which rewards individual reasoning steps rather than just final answers, to improve mathematical problem solving. The approach achieved state-of-the-art performance on mathematical benchmarks by incentivizing correct intermediate steps in reasoning chains. This method also improves model alignment by ensuring the generated reasoning is endorsed by humans rather than just producing correct outputs through potentially opaque means.
Why it matters
We've trained a model to achieve a new state-of-the-art in mathematical problem solving by rewarding each correct step of reasoning (“process supervision”) instead of simply rewarding the correct final answer (“outcome supervision”). In addition to boosting performance relative to outcome supervision, process supervision also has an important alignment benefit: it directly trains the model to produce a chain-of-thought that is endorsed by humans.