Inside VAKRA: Reasoning, Tool Use, and Failure Modes of Agents

Hugging Face Blog · 3 months ago

VAKRA is a benchmark for evaluating AI agents on multi-step reasoning tasks in enterprise environments, combining API interactions with document retrieval across four capability categories. The benchmark contains 5,187 test instances across 62 domains with over 8,000 locally hosted APIs, requiring agents to complete reasoning chains of 3-7 steps that combine structured and unstructured information retrieval. Evaluation is execution-centric, assessing both the correctness of final answers and the validity of the tool-call sequences used to arrive at them.