AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP

arXiv cs.AI · 18 hours ago

AgentCheck is an open-source workbench that helps developers test and fix LLM agent failures by reproducing tool failures, testing mitigations, and confirming fixes work before deployment. The system evaluated five agents across 120 scenarios, with the strongest agent passing 105 scenarios and the weakest passing 77, revealing that failures often manifest as silent misuse of incorrect tool outputs rather than crashes. A retry mitigation improved timeout error handling from 30% success to 100% on the weakest agent, though stale-data faults remained problematic at 3-4 out of 10 cases.