ARC Prize test results for Astra on challenging evaluation setups
ARC Prize ● Covered by 5 sources
Astra hit 62.7% on ARC-AGI-3 with one setup and 99.9% with another. It also beat humans on action efficiency on 96% of levels.
Based on reporting by ARC Prize — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
OpenAI’s Astra just posted two very different results on ARC-AGI-3, and the gap between them says as much about the benchmark as it does about the model. Under the Standard harness, Astra reached 62.7% on the Semi-Private set for $26,098. With the Provider Adapter harness, the same model climbed to 99.9% for $18,817.
ARC-AGI-3 is built around novel, abstract, turn-based environments. The point is not whether a model can guess its way through a puzzle. It’s whether it can explore, infer the goal, build a model of the world, and then plan and correct itself without being spoon-fed instructions. Human participants can solve 100% of the environments, and the benchmark is meant to measure the remaining gap between AI and that kind of flexibility.
What stood out in Astra’s runs was how quickly it turned weird little game mechanics into something compact and usable. It kept notes, but not in a loose human sense. The model developed its own shorthand for objects, coordinates, rules, and unfinished plans, then used that code-like system to track state and sequence actions. In one example, that notation compressed the game into a few symbols that captured where things were, how they interacted, and what should happen next.
There was also a striking efficiency story. ARC’s team tested about 500 people from the general public to set a baseline, then compared Astra against the median action count of humans who finished each level. In the Provider Adapter setup, Astra used fewer actions than that human baseline on 96.0% of levels and averaged 51.7% fewer actions per level. In other words, on this benchmark, it wasn’t just solving more — it was often doing it with less wandering around than people did.
The third thread is tooling. In the PRO-LONG harness, Astra could run custom code, and it built game-specific helpers: parsers, state models, search tools, planners, and persistent notes. For a maze-like game with guards, it even split its work into small files such as maze_solver.py, combat_solver.py, patrol_solver.py, and sync_state.py. That’s useful, but it’s also a different kind of test. The ARC team is careful to separate pure model performance from model-plus-tools performance, and that distinction matters a lot here.
My take — AI-written commentary, not fact-checked reporting
This is the sort of result that makes benchmark debates annoying in a useful way. A provider-managed setup can make a model look dramatically better, which is exactly why the label matters and why “best score” without context is mostly theater. The real lesson is simpler: once a model gets the rules, it can act weirdly human about efficiency — and that’s the part everyone should watch.
Read more about this at: ARC Prize
