Introducing the SWE-Lancer benchmark

OpenAI Blog · 1 year ago

Researchers released the SWE-Lancer benchmark to test whether advanced language models can complete real freelance software engineering tasks and earn money on platforms like Upwork. The benchmark includes 316 tasks from actual freelance projects with an average task value of $3,168. If frontier LLMs succeed at these real-world tasks, it would demonstrate they can perform economically valuable work beyond controlled academic settings.