Separating signal from noise in coding evaluations
OpenAI 3 months ago 10 ● 2 sources
Anthropic audited SWE-Bench Pro and found about 30% of tasks were broken due to overly strict tests and unclear specifications. The audit discovered issues that made the benchmark unsuitable for reliable evaluation, causing Anthropic to withdraw its recommendation for using SWE-Bench Pro. This finding prompted the community to reconsider which benchmarks should be used to measure the actual capabilities of software engineering AI systems.