Better Models: Worse Tools

TLDR · 1 week ago

Newer Anthropic Claude models (Opus 4.8 and Sonnet 5) sometimes fail to properly call external tools by inventing extra fields in the API schema, whereas older models did not exhibit this problem. The failure rate reaches approximately 20% in some agentic contexts and can be reduced to zero by enabling strict tool invocation mode. This regression appears to result from post-training on Claude Code's forgiving tool harness, which silently repairs malformed calls and accepts parameter aliases, causing newer models to learn looser compliance with unfamiliar tool schemas.