Rethinking how we measure AI intelligence

Benchmarks are getting stale, and Google DeepMind knows it. Models trained on massive internet scrapes keep hitting near-100% scores on standard tests, which sounds great until you realize it might just mean they've memorized the answers rather than actually reasoning through them. So DeepMind and Kaggle are trying something different: pitting frontier AI models against each other in actual games, where the winner is whoever plays better, not whoever a human judge happens to like more.

The new platform, called Kaggle Game Arena, starts with chess. Eight top models will face off in a single-elimination exhibition on August 5 at 10:30 a.m. Pacific, with real chess experts hosting. But the exhibition is just the show — the real rankings come from a much bigger all-play-all format, running over a hundred matches between every pair of models to get statistically solid results rather than a lucky bracket run.

Why games? Because they strip away ambiguity. There's no debate about who won a chess match. And the format forces models to show off skills that matter well beyond board games: strategic planning, adapting to an opponent who's actively working against you, reasoning several steps ahead. Kate Olszewska and Meg Risdal, the product managers behind this, point out that specialized systems like Stockfish or AlphaZero already crush any general-purpose LLM at these games without breaking a sweat. Today's frontier models aren't built to specialize, so there's a real gap to close — and DeepMind is betting that watching models struggle to close it will reveal more about genuine intelligence than another saturated multiple-choice benchmark ever could.

Everything here is open-source, including the game harnesses that connect each model to the game environment. That's a deliberate transparency move, letting outside researchers verify results instead of trusting a black-box leaderboard. DeepMind has leaned on games before — Atari, AlphaGo, AlphaStar — and famously got a jolt of surprise from AlphaGo's Move 37, a play that stumped human grandmasters. The hope with Game Arena is that similar unexpected strategies might emerge from LLMs under competitive pressure, giving researchers a rare visual, replayable trace of how these models are actually thinking through a problem.

Chess is just the opener. Kaggle plans to add Go, poker, and eventually video games, each testing different flavors of long-horizon planning. The bet is that an ever-expanding, ever-harder set of game environments can keep serving as a meaningful yardstick even as models improve, rather than becoming obsolete the moment scores flatten out near perfect.