AlignEval: Building an App to Make Evals Easy, Fun, and Automated

Eugene Yan spent the better part of a year fighting with LLM evals — the unglamorous but essential work of figuring out whether your AI product is actually any good. The result of that fight is AlignEval, a free web app now live at aligneval.com, integrated with LangSmith and picked up by VentureBeat. It's built to fix a specific, common mistake: teams write elaborate eval criteria before ever looking at their data, then wonder why the evaluator is useless.

Yan's core argument, borrowed partly from a paper called "Who Validates the Validators," is that you cannot define good eval criteria before you've actually seen how the model behaves. Teams that skip this step end up grading for irrelevant things — generic helpfulness scores, spelling errors that almost never happen — or chasing capabilities the model doesn't have yet, like fully autonomous agents back in 2023. Either way, effort gets wasted on defects that don't matter while the real, moderate-frequency failures go unchecked.

AlignEval's fix is a four-step workflow. Upload a CSV with input and output columns — Yan even provides a 50-article sample built from the Factual Inconsistency Benchmark. Then label 20 to 100 rows as pass or fail, nothing fancier, because binary judgments are faster and more consistent than Likert scales or written rationales. He points to DoorDash and the Llama 2 team as prior art for this binary-first approach. Only after you've actually absorbed the data do you write your one-sentence pass/fail criteria and run it through gpt-4o-mini or claude-3-haiku, checking recall, precision, F1 and Cohen's kappa against your own labels.

The more interesting piece is optimization mode, still in early beta. It splits your labeled set into dev and test partitions, runs several trials tweaking the evaluator against the dev set's F1 score, then checks whether those gains hold on the untouched test set. In one example Yan shares, F1 climbs from 0.571 to 0.727 after five trials. He's upfront that with small samples — 25 rows per split — dev and test metrics can diverge wildly, which just means you need more, and more representative, labeled data.

What's oddly charming is how much of the writeup is about building the thing itself. Yan, by his own account a frontend beginner, prototyped the same app five times across FastHTML, Next.js, SvelteKit and FastAPI before settling on a Next.js frontend with a Python/FastAPI backend, leaning on Cursor to answer basic TypeScript questions. He picked the cheapest, smallest models on purpose — gpt-4o-mini and claude-3-haiku — because they're fast and good enough for binary classification, and he chose Railway over a Hetzner VPS purely because deploying updates with one command beat fiddling with infrastructure he didn't need yet.