Introducing the LiveCodeBench Leaderboard - Holistic and Contamination-Free Evaluation of Code LLMs

Coding benchmarks have a cheating problem, and everyone in the field knows it. Standard tests like HumanEval have been floating around online for years, which means there's a decent chance today's frontier models have simply memorized the answers during training. LiveCodeBench, built by researchers from UC Berkeley, MIT, and Cornell, tries to fix that by pulling problems from LeetCode, AtCoder, and CodeForces and tagging each one with its release date. That timestamp lets evaluators run a model only against problems published after its training cutoff, so the score actually reflects generalization rather than recall.

The benchmark also goes wider than the usual pass-or-fail code generation test. It grades models on four separate tasks: writing code from a problem statement, fixing broken code after seeing an error message (self-repair), predicting what a snippet of code will output on a given input, and predicting expected output straight from a written problem description without even seeing the implementation. Each is scored with Pass@1, basically the odds a single attempt gets it right. Together they paint a picture closer to what an actual AI coding agent needs to do: not just spit out a function once, but reason about behavior, catch its own mistakes, and read specs carefully.

The early results are more interesting than a simple leaderboard ranking. GPT-4-Turbo leads most categories, and its edge widens specifically on self-repair, suggesting it's unusually good at digesting compiler feedback and course-correcting. But Claude-3-Opus flips the script on test output prediction, beating GPT-4-Turbo there, which points to stronger raw language reasoning rather than pure code synthesis skill. Mistral-Large shows the same pattern, doing noticeably better on the natural-language-heavy tasks like test output prediction and code execution than on generation itself. None of these models dominate everywhere, and that's the real finding: current coding benchmarks that only measure generation are missing real differences in how models reason.

Hugging Face is hosting the leaderboard and opening submissions through a GitHub repo, where anyone can run the eval suite against a new model with a single command and submit results via a form. The team is also asking for outside collaborators to help extend the benchmark, which given how fast new models ship, seems like the only realistic way to keep a contamination-resistant leaderboard actually current.