SWE-bench Verified

Model ● Covered in 4 stories + Follow

SWE-bench Verified is a human-validated benchmark dataset designed to measure AI models' ability to solve real-world software engineering tasks by filtering the original SWE-bench to remove unreliable test cases. The benchmark has been widely used to evaluate coding agents, with models like a fine-tuned Qwen 32B achieving 59.4% pass@1, though recent analysis indicates the benchmark has become contaminated with training data leakage and methodological flaws that undermine its reliability as an evaluation metric.

Updated 8 August 2026

Specifications

No specifications recorded yet.

Latest developments

A primer on self-improving agent harnesses Substack · 2 months ago · 5

CoderForge-Preview: SOTA open dataset for training efficient coding agents Together AI · 6 months ago · 6

Why we no longer evaluate SWE-bench Verified OpenAI · 6 months ago · 29

Introducing SWE-bench Verified OpenAI · 2 years ago · 52

Timeline

Month Quarter Year

Q3 2026

Development of autonomous agent harness frameworks for improving large language model reliability and performance Feature update

16 Jul A primer on self-improving agent harnesses

Q1 2026

Q3 2024

13 Aug Introducing SWE-bench Verified

Relationships

Products & technology

Derived from SWE-bench · 1 source
HarnessX integrated with this model · 1 source

Every AI story that matters,
in your inbox by 8am.

TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the day in two minutes. Follow companies and topics for alerts, or get the briefing in Slack. Free, no spam, unsubscribe anytime.