AlignEval: Building an App to Make Evals Easy, Fun, and Automated

Eugene Yan · 1 year ago

AlignEval is a web app that simplifies the process of building LLM evaluators by guiding users through data labeling and evaluation criteria definition. The tool operates in four modes: uploading CSV data, binary labeling after 20 samples, evaluation metrics calculation, and optimization mode starting at 50 labeled samples. The app uses small models like gpt-4o-mini and claude-3-haiku to automatically improve evaluators through trial runs on development and test splits.