Google found a way to test Gemini without seeing the questions
The New Stack Amanda Caswell ● Covered by 2 sources
Google tested Gemini without seeing the private questions. That could make AI benchmark scores a lot harder to game.
Based on reporting by The New Stack, Amanda Caswell — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Google DeepMind says it has pulled off what it calls the first double-blind evaluation of a proprietary frontier model. The basic trick is simple to explain and hard to do: Google keeps Gemini’s weights hidden from the evaluators, while the evaluators keep the benchmark questions hidden from Google.
The pilot ran Gemini 2.5 Flash Lite against private tests from MLCommons and the Singapore AI Safety Institute. But the point here isn’t a shiny new score. Google is pushing the method itself, because benchmark leakage has become a real problem as datasets and public tests keep growing.
The company’s report leans on earlier research that found signs of leakage in roughly half of 31 models tested, plus another study from this year that said contamination can especially inflate results for larger models. That matters because a lot of post-training gains, especially in coding, have already been under suspicion. If the model has seen the test before, the score is doing less work than it claims.
The setup uses Google Cloud Confidential Space, NVIDIA’s H100 Confidential GPU and Intel TDX host memory encryption. Google provides the model weights and inference code; the evaluator provides prompts and evaluation code. Both sets of data go through encrypted connections into the enclave, where the test runs without either side getting access to the other’s protected material. Afterward, the evaluator gets the allowed results and the temporary environment can be torn down.
There’s still plenty of friction. Code inside the enclave can still leak data if it has free network access, so OpenMined’s PySyft is used to approve code in advance and block sensitive parts from making outside connections. The paper says the compute overhead is under 5%, and points more to legal agreements and code review as the real pain points. Google also can’t make the whole thing fully trustless: the guest OS is open source and externally validated, but individual builds rely on private signing keys, and Google’s own services help sign and verify the attestation report.
The pilot used a single H100 80GB Confidential GPU, but the researchers are already looking at clusters of H100 and B200 chips linked by encrypted connections for models that won’t fit on one GPU. If this catches on, benchmark scores could mean a little more than they do now. Not everything, just more than a polite shrug.
My take — AI-written commentary, not fact-checked reporting
This is the kind of boring infrastructure AI badly needs. The industry has spent years turning benchmarks into theater, then acting surprised when the audience noticed the trap door. Closed models want private tests without private trust, and this is at least a serious attempt to stop the magic show from stealing the watch.
Read more about this at: The New Stack