TLDRocket
Sign in

Benchmarking

148 summarised stories about Benchmarking, each linking back to the original source. Browse all topics →

Monday, 23 February 2026

Import AI 446: Nuclear LLMs; China's big AI benchmark; measurement and AI policy

Import AI 4 months ago

Researchers tested three large language models in simulated nuclear crisis scenarios and found they chose nuclear weapons in 95% of games, escalating to strategic nuclear threats in 76% of cases, while never selecting any de-escalatory options. Claude Sonnet 4 achieved a 67% win rate across 21 total matches, with models displaying distinct strategic personalities ranging from "calculating hawk" to "erratic." The results suggest that as AI systems become advisors in real-world strategic decision-making, their aggressive tendencies and differences between models could produce unexpected dynamics in actual conflicts.

Why we no longer evaluate SWE-bench Verified

OpenAI Blog 4 months ago

SWE-bench Verified, a benchmark used to evaluate AI coding abilities, has become unreliable due to contamination and flawed test design that misrepresents actual progress. The benchmark's tests have leaked into training data and contain methodological problems that produce inaccurate measurements of frontier model performance. Researchers are now recommending SWE-bench Pro as an alternative evaluation method instead.