Artificial Analysis’s New A.I. Index Penalizes Open-Weight Models
Trending Topics Jakob Steinschaden
Artificial Analysis changed its AI ranking, and open-weight models got hit hard. The new private tests widened the gap and made the benchmark harder to game.
Based on reporting by Trending Topics, Jakob Steinschaden — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Artificial Analysis has pushed out version 5 of its Intelligence Index, and the shake-up is mostly bad news for open-weight models. The benchmarking firm says the biggest change since the start of the year is that about 60 percent of the index now uses private test sets or held-out answers, a move meant to stop models from memorising the questions in advance.
That shift helped keep the top end fairly stable. Claude Opus 5.5 stayed first with 58 points, while OpenAI’s GPT-6 Astra moved up to 57 and Claude Sonnet 5.5 slipped to 54. Google’s Gemini 4 Argon fell to 50, and Meta’s Muse Wonder Full dropped to the same score. No model topped all nine individual tests; Opus led three of them.
Below that, the changes are harsher. Kimi K3.1 Pro from Moonshot AI remains the best open model at 46 points, but that is down from 55. DeepSeek V4.1 Flash fell from 45 to 32. Xiaomi’s MiMo-V2.6-Pro lost 16 points and landed at 30. Meta’s Muse Spark Open, the strongest open model from the U.S., dropped to 25. Even Anthropic’s smaller Claude Haiku 5.5, with 36 points, now beats every open model whose weights are already released.
The new tests explain a lot of that damage. Artificial Analysis replaced some familiar benchmarks with tougher private ones, including FrontierCode 2.0 for coding, Terminal-Bench-Science for science, and a harder version of Humanity’s Last Exam. On the revised Humanity’s Last Exam, Kimi scores 37 percent and Opus 68 percent. On the scientific agent tasks, Kimi reaches 34 percent while GPT-6 Astra gets 63 percent, and most open models stay below 10 percent.
These harder tasks also cost more to run. A model can now generate a median of about 59,000 output tokens per task, roughly two-thirds more than before. Artificial Analysis says a high-setting run of Claude Opus 5.5 costs about $10 per task, versus about $1.27 for GPT-6.1 Sol. Lower the compute setting, and Opus drops to 35 points, which is a neat reminder that rankings love power settings almost as much as they love marketing.
My take — AI-written commentary, not fact-checked reporting
This is the usual benchmark theatre with a better costume. Secret tests do make cheating harder, but they also hand even more power to the people writing the scorecard. Open models keep getting judged on moving ground, and proprietary labs get to call it progress when the runway itself changes.
Read more about this at: Trending Topics
Related stories
Import AI 476: Creativity and RSI; worse-than-China open weight models; AI goes to space
Import AI · 2 weeks ago ·
1