Data Machina #254

Data Machina · 2 years ago

Princeton Language & Intelligence released SWE-bench, a benchmark for evaluating AI coding agents on their ability to fix real GitHub repository issues, revealing that current AI agents perform poorly on the task. The leading model, Amazon Q Developer Agent, successfully solved only 13.8% of 2294 tasks, while the open-source OpenDevin achieved the highest benchmark score at 21%. These results demonstrate that despite multiple competing approaches including Devin, Devika, and GPT-Engineer, AI coding agents remain far from ready for production-scale legacy code migration and autonomous software engineering work.