Is it agentic enough? Benchmarking open models on your own tooling

Hugging Face Blog · 4 weeks ago

Researchers benchmarked how well different language models can use the transformers library by measuring not just whether they got the right answer, but how much effort it took them to get there. The evaluation framework tested each task across three different tool access levels (bare install, cloned source code, or packaged skill) and found that adding a CLI reduced median task completion time but increased token consumption by 60% due to agents reading documentation. The results show that library design significantly affects agent efficiency—smaller models benefited more from curated documentation while larger models sometimes performed better with full source code access.