Introducing the SWE-Lancer benchmark
OpenAI Blog
Researchers released the SWE-Lancer benchmark to test whether advanced language models can complete real freelance software engineering tasks and earn money on platforms like Upwork. The benchmark includes 316 tasks from actual freelance projects with an average task value of $3,168. If frontier LLMs succeed at these real-world tasks, it would demonstrate they can perform economically valuable work beyond controlled academic settings.
Why it matters
Can frontier LLMs earn $1 million from real-world freelance software engineering?
