Benchmarking showed that different evaluation harnesses for the same AI coding agents produced widely different token usage and costs
Benchmark result Provisional 76% confidence first seen
Two articles report on benchmarking of AI coding agents (using an identical underlying model) where changes in the evaluation harness led to very large differences in token consumption per solved task. The coverage argues this shifts optimization toward harness prompt/context overhead and cache-hit behavior, and recommends tracking cost per verified outcome and failure-focused harness logging rather than raw token counts.