The Sequence AI of the Week #895: OpenAI's Show Us Where Coding Evals Break

TheSequence · 6 days ago

OpenAI audited SWE-Bench Pro, a coding evaluation benchmark, and found that approximately 30 percent of its 731 public tasks contain defects such as rejecting correct solutions or accepting incomplete ones. OpenAI's agent-assisted review labeled 27.4 percent of tasks as defective while independent software engineers identified 34.1 percent as problematic. OpenAI withdrew its earlier recommendation that the field adopt SWE-Bench Pro as a standard evaluation tool due to these validity issues.