Separating signal from noise in coding evaluations

OpenAI Blog · 1 week ago

OpenAI's analysis identified flaws in SWE-Bench Pro, a widely-used benchmark for evaluating coding AI systems, questioning whether its results accurately reflect model performance. The benchmark contains ambiguous test cases and inconsistent evaluation criteria that affect reliability of the results. This finding prompts developers to scrutinize existing coding benchmarks more carefully and potentially revise evaluation methodologies before trusting benchmark rankings.