Open LLM Leaderboard: DROP deep dive
Hugging Face
Hugging Face pulled the DROP benchmark from its Open LLM Leaderboard after finding the scoring was basically broken. Nearly every model was tanking the test not because they're dumb, but because the grading script was cutting off answers early.
Based on reporting by Hugging Face — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Three weeks after Hugging Face added DROP, Winogrande and GSM8k to its Open LLM Leaderboard, something looked off. Most pretrained models were scoring under 10 out of 100 on DROP's f1 metric, even ones that were clearly strong performers everywhere else. When the team plotted DROP scores against the leaderboard's usual average of ARC, HellaSwag, TruthfulQA and MMLU, only a handful of models tracked the expected trend. The rest just clustered near a score of 5, regardless of how good they actually were.
The first bug was in how answers got normalized before comparison. If a model generated a correct numeric answer like "10" but it was immediately followed by a line break instead of a plain space, the normalization code failed to recognize it as a number at all. So the gold answer "10" got converted to "10.0" while the model's technically-correct "10" stayed a plain string, and the two never matched during the bag-of-words comparison. A model could nail the answer and still get marked wrong purely because of formatting.
Working with the Zeno team, Hugging Face found something even messier: not one model got a floating-point answer correct, and stronger models with longer, more natural outputs actually scored worse. Both problems traced back to the same culprit — the period character was being used as the stop token for generation. That meant decimal answers like "12.25" got chopped off after "12," and better models that mimicked the few-shot prompt format kept generating past their real answer into a fake next question, only stopping at the first period they hit later, which padded their output with irrelevant words and tanked their f1 score.
Swapping the stop token to a line break instead of a period fixed a lot of this, at least in testing. Re-scoring existing outputs by splitting on the first newline produced scores that correlated much better with overall model quality. But it's an imperfect patch — it doesn't rescue answers that were already truncated by an errant period, so floating-point answers are still a mess. Estimates suggest more than half of all DROP examples would need to be rerun to get trustworthy numbers, and DROP evaluation alone reportedly ate a huge chunk of the eight years of cumulative GPU time spent on the leaderboard's last full refresh.
After digging through it with the EleutherAI Harness team, Hugging Face concluded the implementation faithfully followed DROP's original scoring code — which means the benchmark itself has been quietly broken for a while, not just this one integration. Rather than keep publishing misleading numbers, the leaderboard is dropping DROP entirely until someone builds a fixed version.
My take — AI-written commentary, not fact-checked reporting
This is exactly the kind of thing that never gets caught behind closed doors — a benchmark everyone quietly trusted for years, unraveled because a bunch of outside people actually looked at the outputs instead of just the leaderboard. Closed labs would have buried this or never noticed, because nobody outside gets to poke at the eval code. Open tooling isn't just a nicety here, it's the entire reason this bug got found at all.
Read more about this at: Hugging Face