Introducing the SWE-Lancer benchmark

OpenAI's newest benchmark skips the usual academic coding puzzles and goes straight for something messier: real freelance software engineering work, the kind posted on job boards with actual dollar values attached. SWE-Lancer, as they call it, poses one blunt question — could a frontier language model take on these gigs and walk away with a payout, the way a human contractor would?

The setup matters here. Instead of grading models on toy problems with clean right answers, OpenAI pulled tasks that carry real-world stakes, tallying up to a million dollars in potential earnings across the set. That's a deliberate choice. Freelance engineering work is full of ambiguity, half-written specs, legacy code nobody wants to touch, and clients who don't always know what they're asking for. It's a much rougher test than anything on a typical leaderboard.

What's notable isn't just the framing but the honesty of it. OpenAI isn't pretending its models have already cracked this. The benchmark exists because nobody currently knows how close frontier systems are to functioning like an actual paid engineer, end to end — writing code, fixing bugs, navigating vague requirements, and producing something a client would accept and pay for. That's a very different bar than passing a coding interview question.

This fits a pattern we've seen from OpenAI and rivals over the past year: benchmarks are quietly shifting away from abstract reasoning tests toward economically grounded ones. Instead of asking whether a model can solve a riddle, the industry is starting to ask whether it can do a job that someone would otherwise pay a human to do. SWE-Lancer is one of the more direct attempts at putting a price tag on that question.