The Sequence AI of the Week #895: OpenAI's Show Us Where Coding Evals Break

Substack · 3 months ago · 12

OpenAI audited SWE-Bench Pro, a coding evaluation benchmark, and found that approximately 30 percent of its 731 public tasks contain defects such as rejecting correct solutions or accepting incomplete ones. OpenAI's agent-assisted review labeled 27.4 percent of tasks as defective while independent software engineers identified 34.1 percent as problematic. OpenAI withdrew its earlier recommendation that the field adopt SWE-Bench Pro as a standard evaluation tool due to these validity issues.