Coding agents can be evaluated. We just have to evaluate the work.

A software factory provider told the writer coding agents can’t be evaluated. The case sounded neat: software work is messy, requirements are incomplete, repositories are full of old decisions, and two engineers can solve the same problem in different but valid ways. A coding agent can also fail one run and pass the next. None of that is fake. It just doesn’t prove the thing is unevaluable.

The argument here is that the unit under test isn’t just the model. A coding agent is a bundle: model, harness, tools, repository context, instructions, permissions, execution environment, and a feedback loop. Move one piece and the result can change a lot. That’s why raw benchmark scores are so easy to misread. They often say less about the model itself than about a very specific setup, with a specific token and time budget.

So the evaluation has to look at the repository, not a reference patch. Start from a known state. Give the agent the task and the context it had. Then check the finished code against executable contracts: does it build, do the tests still pass, do hidden tests catch the requested behavior, are APIs and data formats still compatible, do migrations work both ways, are performance limits respected, did it wander outside the allowed scope, did static analysis or security checks flag anything new? That allows more than one valid implementation while still setting a clear bar.

A pass rate alone is too cheap. An agent can weaken an existing test, hard-code a value, or take twenty tries and finally stumble into something correct while burning budget and leaving a mess behind. The piece argues for a scorecard: outcome, change quality, trajectory, human help, economics, and what happens after merge. That’s a more honest view of delivery than a single number ever will be.

The same logic applies when the task is vague. If the ask is “make onboarding better” or “build the new billing workflow,” then the evaluation should include interaction. The agent should be checked on whether it asks useful questions, notices ambiguity, uses answers properly, and avoids inventing requirements. Newer benchmarks like ICAE-Bench and Dialogue SWE-Bench are already moving that way. The basic point is simple: ambiguity makes evaluation harder. It doesn’t make it impossible.