Introducing the LiveCodeBench Leaderboard - Holistic and Contamination-Free Evaluation of Code LLMs

Hugging Face · 2 years ago · 28

Researchers from UC Berkeley, MIT, and Cornell released LiveCodeBench, a new leaderboard for evaluating code-generation capabilities of large language models across four tasks: code generation, self-repair, code execution, and test output prediction. The benchmark collects problems from LeetCode, AtCoder, and CodeForces with annotated release dates, enabling evaluation on problems released after a model's training cutoff to prevent contamination. GPT-4-Turbo performs best on most scenarios, while Claude-3-Opus excels at test output prediction and Mistral-Large shows stronger performance on natural language reasoning tasks.