AstaBench update: New results, plus adoption from industry

AI2 just dropped a fresh batch of scores for AstaBench, its open benchmark that tries to answer a blunt question: can AI agents actually do science, not just talk about it. The team ran GPT-5.5, GPT-5.4, Claude Opus 4.7 and 4.6, Claude Sonnet 4.6, and Gemini 3.1 Pro Preview through more than 2,400 research problems spanning literature search, coding, data analysis, and full end-to-end discovery.

Claude Opus 4.7 tops the leaderboard at 58.0% overall, but it also costs $3.54 per problem, roughly double what GPT-5.5 costs to hit 52.9%. That's the real story here: performance and price are climbing together, and nowhere is that more obvious than in end-to-end discovery, the hardest category, where an agent has to carry a research idea all the way through to working code and a written report with zero hand-holding. Opus 4.7 beats Opus 4.6 by 10 points there, but needs 54% more steps and 65% more money to do it. Some of that cost bump traces to Opus 4.7's new tokenizer, which AI2 notes can inflate token counts by up to 1.35x for identical text.

GPT-5.5 is arguably the more interesting result. It comes within about five points of the top Claude model while spending less than half as much per problem, and it actually leads on code execution, data analysis, and edges out Claude on literature understanding. But it still lags on end-to-end discovery, which is the crux of the whole benchmark. Being good at the individual pieces — writing code, parsing a dataset, summarizing papers — doesn't automatically add up to finishing a real research task. AI2's original test last August found its best agent completed only 3% of end-to-end tasks perfectly, often stalling out after nailing 60-70% of the steps. That gap hasn't closed; it's just gotten pricier to attempt closing it.

AI2 also tightened its scoring. The models grading ScholarQA-CS2 and end-to-end discovery got swapped for newer versions after the old ones were deprecated, and the new end-to-end scorer is stricter about catching fabricated results and placeholder code that used to slip through. That means some of the score shifts reflect harder grading, not just weaker models, which is a useful reminder that benchmark numbers move for more than one reason.

Beyond the leaderboard, AstaBench is picking up outside users. The UK AI Security Institute's Inspect Evals collection, built with Arcadia Impact, is folding AstaBench in, and General Reasoning has turned one of its tasks into a training environment on its OpenReward platform. Elicit, SciSpace, Distyl AI, and EvoScientist have submitted their own agents too. AI2 built the whole thing open-source specifically so outsiders could pile on and cross-check the results, and it's starting to look like that bet is paying off.