What We Learned by Reproducing 2,200 papers from ICML

In July, Hugging Face turned ICML 2026 into a giant open reproduction test. More than 1,200 community members joined in, each bringing their own coding agent, and over 19 days they produced 6,816 logbooks covering 2,226 papers — about a third of the conference.

The setup was built to make the auditing itself auditable. Participants picked from the 6,341 accepted papers, worked from extracted claims instead of raw PDFs, and published everything they ran: code, artifacts, write-ups, and sometimes full agent traces. A judge based on an open-weights model, GLM-5.2, then re-read every logbook and assigned claim-level verdicts: verified, falsified, toy, or inconclusive.

The scale is the point. ICML 2026 drew 23,918 submissions and accepted 6,352 papers, roughly twice the previous year. Reviewers, meanwhile, are still human volunteers with limited time and limited patience for proof checking. The post argues that agents can now help close that gap, not by replacing review but by making large-scale checking possible at all.

And the results were messy in the way real science usually is. Of the papers examined, 1,103 had at least one claim independently verified. Another 496 had at least one claim falsified or contested, including 49 where every claim fell apart. There were also 242 papers where different teams reached opposite conclusions on the same claims, which is a good reminder that reproducibility is not a checkbox.

Some papers survived very well. One on flat minima was reproduced by 20 independent teams, with 12 verifying every claim. Another, about LLM judges and adversarial robustness, saw 14 of 17 logbooks verify everything. Others did not. The paging paper from the introduction was found to have a robustness term growing like log k rather than staying constant, and one paper on attention and Frank-Wolfe had counterexamples show up at several different step counts. In another case, a paper’s theory used reverse KL, while the released code defaulted to forward KL and produced the reported results.

The sharpest lesson is that agents are useful, but not enough. They got trapped in loops, missed scale-dependent behavior, and sometimes built a whole “falsification” on a units mistake. The strongest reproductions came when a human kept steering: redirecting the agent, challenging assumptions, or deciding the question itself was wrong before the compute bill got silly. The best use of people here is not as typists. It’s as editors for machine effort.