Product Evals in Three Simple Steps

Eugene Yan just published the kind of practical guide that gets bookmarked by every ML engineer tired of reinventing eval pipelines from scratch. His pitch is refreshingly unglamorous: skip the fancy scoring systems and Likert scales, and build evals in three steps — label a small dataset, align an LLM judge to those labels, then wire the whole thing into an experiment harness that runs automatically on every config change.

The labeling advice cuts against a lot of instinct in the field. Yan argues binary pass/fail or win/lose/tie labels beat 1-5 scales almost every time, because humans can't consistently tell a 3 from a 4, and if humans can't do it reliably, an LLM judge trained on that fuzzy signal won't either. He wants 50 to 100 failure cases out of a 200-plus sample set, and he's specifically skeptical of synthetic failures generated by prompting a strong model to misbehave — those tend to be either cartoonishly bad or too subtle, missing the messy organic failures real users actually trigger. His fix: run weaker models to generate outputs naturally, since they fail in realistic ways, and use active learning to hunt down real failures in unlabeled production data instead of eyeballing everything.

On the evaluator side, Yan pushes hard against what he calls the