What's the largest software project AI can complete on its own?

Most coding benchmarks ask AI to patch a bug or bolt on a feature — the kind of thing that takes a human an afternoon. MirrorCode, built by an AI research group alongside METR, throws out that scale entirely. The task: hand a model zero access to a program's original code and make it rebuild the whole thing from nothing, then check whether it behaves identically on tests it's never seen, including some held out specifically to catch cheating.

The 25 target programs aren't toy examples. They span Unix utilities, bioinformatics tools, compression libraries, interpreters, and cryptographic code. One of the gnarlier ones, gotree, is a bioinformatics toolkit weighing in at roughly 16,000 lines of Go across more than 40 commands. The researchers estimate a skilled human engineer working solo would need somewhere between two and seventeen weeks to reproduce it. Claude Opus 4.7 did it in 14 hours, for $251, and passed 2,000 of 2,001 tests — missing only a single edge case involving date-annotation handling.

That's the headline number, but the more interesting detail is the money. Most software benchmarks cap inference spending at $1 to $10 per task, which is absurd when you consider some of these jobs would occupy a human for months. MirrorCode instead lets models run wild: one of the largest tasks burned through $2,600 in compute and had an AI grinding away, unsupervised, for 19 straight days. That's not a benchmark score — that's closer to an employment contract.

The obvious worry is contamination. Since these are real open-source projects, there's a decent chance the models memorized chunks of them during training, which would make the results look better than they are. The team ran a memorization screen and found that AI models that showed signs of having memorized a program's structure tended to fail at reimplementing it faithfully, while models without that signal still managed to solve tasks correctly — evidence, though not proof, that raw capability rather than recall is doing the work.

MirrorCode isn't close to being solved. The team's own leaderboard tracks 30 tasks across two languages, usually Go paired with the far less common Ada, giving each attempt a budget of 10 billion tokens and a full week. Twenty-two of the 25 target programs are open-sourced already, with three held back as a private test set so scores can't just be gamed by studying the answer key in advance.