IBM and UC Berkeley Diagnose Why Enterprise Agents Fail Using IT-Bench and MAST
Hugging Face Blog 5 months ago
IBM and UC Berkeley created MAST, a diagnostic framework that categorizes why AI agents fail in IT automation tasks rather than just reporting success rates, and applied it to analyze 310 execution traces across three language models. Gemini-3-Flash averaged 2.6 failure modes per failed trace while GPT-OSS-120B averaged 5.3, showing that smaller open-source models suffer from cascading failures that compound over time. The analysis revealed that incorrect verification (agents declaring success without checking results) is the strongest failure predictor, enabling developers to deploy targeted fixes like external verification gates instead of blind prompt engineering.