A shared playbook for trustworthy third party evaluations

OpenAI has put out a new playbook aimed at the people and organizations who evaluate its frontier models from the outside. The document walks through how to judge a model's raw capabilities, how to check whether its safeguards actually hold up under pressure, and how to make sure the tests themselves are measuring something real rather than just producing a nice-looking score.

That last point is the sneaky one. Anyone who has watched benchmark scores climb while real-world reliability barely budges knows that a test can look rigorous and still tell you almost nothing useful. OpenAI's framing pushes evaluators to ask whether a given test actually predicts behavior in the wild, not just whether the model clears some threshold in a sandbox.

The timing is not random. Regulators, journalists, and researchers have been leaning harder on third-party audits as the main check on companies that build and grade their own systems. OpenAI, Anthropic, and Google DeepMind have all faced pointed questions about why outside evaluators get limited access, limited time, or incomplete model versions before a launch. A shared playbook, at least on paper, gives independent testers something to point to when they push back on those constraints.

What the guidance does not solve is the access problem itself. Standards for how to run an evaluation matter far less if the company being evaluated still controls who gets in the room, what version of the model they see, and how much time they get before release. OpenAI describing best practices for others to follow is useful, but it is not the same as OpenAI submitting to them.