"It's Hard to Eval" Is a Product Smell

TLDR · 2 weeks ago

A product designer argues that when AI systems are hard to evaluate, it reflects poor product design rather than an evaluation problem, and demonstrates how to restructure products to make verification easier by showing working, sources, and incremental changes rather than opaque final outputs. The examples span data agents showing 50% more detail through notebooks, PE lesson planners anchored to vetted templates with diffs, and medical report generators that surface contradictions and source citations before the final document. Redesigning products for verifiability reduces user friction, enables better automated testing, and creates feedback loops that improve quality over time.