Open LLM Leaderboard: DROP deep dive

Hugging Face · 2 years ago · 25

The Open LLM Leaderboard discovered critical flaws in its DROP benchmark implementation, where most models scored below 10 out of 100 on the f1-score metric despite appearing capable on other benchmarks. The investigation identified two main issues: the normalization step failed when numbers were followed by non-space whitespace characters, and using a period as the stop token prevented models from completing floating-point answers and generated extraneous text. The benchmark has been removed from the leaderboard pending development of a corrected evaluation implementation, as fixing the issues would require rerunning over 50 percent of test cases.