TLDRocket
Sign in

AI Benchmarking

292 summarised stories about AI Benchmarking, each linking back to the original source. Browse all topics →

+ Follow this topic

Thursday, 27 August 2026

Google found a way to test Gemini without seeing the questions

The New Stack 1 month ago 17 ● 2 sources

Google DeepMind demonstrated a double-blind evaluation for Gemini where evaluators cannot see Gemini’s weights and Gemini cannot see the test questions. The pilot evaluated Gemini 2.5 Flash Lite against private MLCommons and Singapore AI Safety Institute benchmarks using an NVIDIA H100 Confidential GPU with Intel TDX and Confidential Space, with evaluation overhead reported as under 5%. This shifts focus from publishing a benchmark score to showing how to run tests with reduced benchmark leakage risk by keeping both sides’ data protected inside an encrypted enclave.

“Posterity will find it ludicrous”: Sai agent hits 73% on OSWorld 2.0 performing routine (but necessary) work

The New Stack 1 month ago 46

Simular’s computer agent Sai achieved a 73% success rate on the OSWorld 2.0 108-task benchmark. The benchmark is for tasks that typically take skilled humans more than 1 hour to complete. Simular claims Sai delivers better outcome-to-cost tradeoffs (about 2/3 the cost of competing agent results) by using neuro-symbolic planning that reduces model calls and improves caching.

Piloting the world's first double-blind AI evaluations

Google 1 month ago 17 ● 2 sources

Google introduced the world’s first double-blind evaluation of a proprietary frontier AI model, using a cryptographic “box” so external evaluators’ benchmarks can’t be used for later optimization. On August 27, 2026, Google said it would test a Gemini Flash Lite model against confidential benchmarks with partners including the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons. This adds cryptographic safeguards to reduce benchmark contamination and improve trust in the measured model results.

Google Research Introduces GlucoFM: A 0.72M-Parameter Dual-Stream Foundation Model for Continuous Glucose Monitoring

MarkTechPost 1 month ago 51 ● 2 sources

Google Research and UNSW Sydney introduced GlucoFM, a self-supervised foundation model that splits continuous glucose monitoring traces into separate slow state and transient event streams. It has 0.72M trainable parameters and reached 58.8 task-averaged PR-AUC across 14 cohort–task evaluations. The work is a research prototype with no regulatory clearance and no public checkpoint as of 26 August 2026, but the code/reproducibility plan and CPU/24-hour inference recipe change deployment from model use to a reproducible training workflow.

Z.ai open-sources ‘Ox Alpha’ model as GLM-5.3-Flash

SiliconANGLE 1 month ago 31 ● 2 sources

Z.ai released the code for GLM-5.3-Flash after its earlier Ox Alpha debut. The model activates 18 billion of its 320 billion parameters and is described as costing 10 times less to run than the prior generation. With sparse and linear attention changes plus mHC training, it reduces RAM and computation overhead while targeting strong benchmark performance and making weights available on Hugging Face.

The daily briefing

Every AI story that matters, in your inbox by 8am.

TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the day in two minutes. Follow companies and topics for alerts, or get the briefing in Slack. Free, no spam, unsubscribe anytime.