AlignEval: Building an App to Make Evals Easy, Fun, and Automated
Eugene Yan
AlignEval is a web app that simplifies the process of building LLM evaluators by guiding users through data labeling and evaluation criteria definition. The tool operates in four modes: uploading CSV data, binary labeling after 20 samples, evaluation metrics calculation, and optimization mode starting at 50 labeled samples. The app uses small models like gpt-4o-mini and claude-3-haiku to automatically improve evaluators through trial runs on development and test splits.
Why it matters
Look at and label your data, build and evaluate your LLM-evaluator, and optimize it against your labels.
