Prover-Verifier Games improve legibility of language model outputs

OpenAI's latest paper borrows a game-theory trick called prover-verifier games to make language models write answers that aren't just correct, they're checkable. The test bed is GSM8K, the well-worn benchmark of grade-school word problems about apples, buses, and allowance money, chosen because right and wrong answers are easy to define.

The setup pits a large "prover" model against a much smaller "verifier." Over repeated rounds, the prover gets rewarded when the verifier can correctly judge its solution. There's also a "sneaky" prover, deliberately trained to produce wrong answers that still look convincing, which forces the verifier to get better at catching subtle mistakes instead of being swayed by confident-sounding text.

After several rounds of this back-and-forth, the prover started writing step-by-step solutions that human graders, who had no part in the training loop, could follow faster and judge more accurately. Legibility scores rose noticeably while raw accuracy dipped only slightly, a trade most people would take gladly if it means actually understanding why a model reached an answer instead of just trusting the output.

The bigger point is about what happens once models get smarter than the humans or smaller systems meant to check them. Nobody can fully audit a superhuman model's raw reasoning by eye, so building an incentive at training time to stay legible rather than merely persuasive could matter a lot down the line. This is a small, measurable step on that problem, run on an actual benchmark rather than a hypothetical about future AGI.

It's still narrow. GSM8K is elementary arithmetic, the human evaluators graded quickly and weren't infallible, and legibility as tested here is one specific, quantifiable trait rather than a solution to alignment writ large. But it's the kind of unglamorous, testable progress that's easy to overlook next to bigger model releases.