All of AI benchmarking at your fingertips
IBM Research
Researchers from IBM, Hugging Face, and other institutions launched EveryEvalEver, a standardized database and JSON format for documenting AI model benchmark results to address inconsistencies in how evaluation scores are reported across the industry. The platform currently aggregates 22,000 model results across 2,200 benchmarks translated from 31 different evaluation formats, revealing that identical evaluations can produce scores varying by as much as 20 percentage points due to different harnesses and undocumented settings. By establishing a common language for benchmarking metadata and making prior results reusable, the project aims to reduce the $370,000 cost of reproducing aggregated evaluations and improve transparency in model comparisons.
Why it matters
IBM is part of a global team trying to make AI benchmarking results easier to compare, replicate, and reuse.
