Open-world evaluations for measuring frontier AI capabilities

AI Snake Oil · 3 months ago

Researchers introduced open-world evaluations, a new method for testing AI capabilities in complex real-world tasks beyond standard benchmarks, and launched CRUX, a collaboration of 17 researchers that successfully tasked an AI agent with building and publishing an iOS app to the App Store. In CRUX's first experiment, the agent completed the task after two errors, one requiring manual intervention, with the entire process costing approximately $1,000. This approach aims to provide early warnings about emerging AI capabilities and identify blind spots in existing benchmarks before such abilities become widespread.