"It's Hard to Eval" Is a Product Smell

Jason Liu has spent three years building AI evals for a living, and he keeps hearing the same excuse from founders: our product is just hard to eval. His pushback is blunt. If your AI's output is hard for you to score, it's probably hard for your actual users to check too, and that's not an evals problem, it's a design problem hiding behind one.

His first example is the internal data agent almost every company seems to be building right now, the kind that answers "what was net revenue for Product A last quarter" with a single number and nothing else. There's nothing to verify in a lone dollar figure. Liu's fix borrows from how data scientists actually audit numbers themselves: confirm the metric definition, check a related figure that should move in tandem, break the total down by region or time, and read the underlying SQL. He sketches a two-tab interface, chat on one side and a full notebook on the other, where the agent shows its assumptions, its queries, and an explicit list of what it couldn't confirm. Hex, he notes, already ships something close to this, pairing chat with live notebooks rather than hiding the work behind a single answer box.

The second case is smaller stakes but the same disease. A founder building an AI tool that writes PE lesson plans for K-12 teachers asked Liu how to score the plans it generated from scratch. He flipped the question: what actually makes a teacher trust a lesson plan? Usually, that another teacher already uses it. So instead of generating a plan from nothing, the redesigned tool starts from a vetted plan, shows who uses it and how often, and presents only the handful of edits needed to fit this teacher's class size, equipment, and location. The teacher reviews two tweaked drills instead of judging fifty minutes of content cold, and evaluation gets easier for the same reason — there's far less surface area to check.

The workers'-comp example raises the stakes considerably. The tool in question reads a patient's entire medical chart and writes a fifty-page expert opinion report that a doctor has to sign off on. As built, verifying that report meant re-reading the whole chart, which takes about as long as writing the thing yourself. Liu's proposed fix turns the tool into a research assistant rather than a report generator: it pulls facts from each record with page-linked citations, flags contradictions between exams, and calls out open questions the chart never resolves. Only after the doctor has worked through those flagged items does the tool assemble the final report from what's already been checked.

Across all three, Liu lands on the same underlying fix: provenance. Show where each claim in the output came from, let users check small scoped pieces instead of a monolithic result, and use progressive disclosure so sources don't bury the person reading them. Do that first, he argues, and the evals basically build themselves — because now there are discrete, gradeable units instead of one giant unverifiable blob.