The Agent Said It Was Done. The Database Disagreed.

Hugging Face · 1 week ago · 31

Microsoft ThinkingBox grades AI agents by the database state and side effects they leave behind, then tests whether they can meet the required end state 20 times in a row despite gaps between tool-call traces and backend outcomes. In a 507-task evaluation with 20 repeated runs each, 79,853 attempts (out of 121,680) failed executable checks, and 77.61% of those still wrote wrong field values. The result is a benchmark that shifts comparisons from how often an agent appears to work to how reliably it updates the correct records and avoids unintended effects, with models ranked by consistency cost rather than pass@1 alone.