olmo-eval: An evaluation workbench for the model development loop

Allen Institute (AI2) · 1 month ago

Researchers at AI2 released olmo-eval, an evaluation workbench designed for iterative LLM development that extends their earlier OLMES benchmarking standard. The tool enables developers to add benchmarks with minimal code, run evaluations across model checkpoints in flexible ways, and compare results question-by-question to detect real improvements rather than noise. Unlike existing frameworks that evaluate finished models, olmo-eval is built to keep pace with continuous model changes during development, supporting tool use, multi-turn interactions, and modular component swapping.