Can we trust LLMs to judge whether a research idea is novel?

The Allen Institute for AI has released a study called Old Ideas, Novel Problems, aimed at a question that matters more as AI starts helping with research: can large language models reliably tell whether an idea is actually novel? Alongside the paper, AI2 is publishing a benchmark, data, code, and an automated pipeline for building new evaluation sets.

This sits inside a broader set of AI2 projects meant to support scientific discovery. DataVoyager helps researchers explore structured data, AutoDiscovery generates and tests hypotheses, and Theorizer pulls evidence from large bodies of literature into candidate theories. The pitch is straightforward enough: if machines are going to help produce ideas, someone has to check whether those ideas are really new.

That check is what AI2 calls a novelty judge, and the study says those judges are far from dependable. Across six models — GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6 — small changes in setup produced big swings in verdicts. In one case, changing just a few words in a prompt cut GPT-5.4’s accuracy from 93.5% to 40.9% on the same Human+Generated pairs. The model also changed its answer on more than half those pairs.

To build the benchmark, AI2 mined ICLR 2026 peer reviews for explicit comments on originality instead of using acceptance as a shortcut for novelty. It created 154 high-novelty human ideas from accepted papers in the top-rated 10% of their research area, and 145 lower-novelty ideas from rejected papers in the bottom 10%. An annotator agreed with the review-based labels in 47 of 50 spot checks, and none of the disagreements would have flipped a paper from praising to disputing novelty, or the other way around.

AI2 then tested a second setting where those human ideas were paired against 154 ideas from a simple LLM generator that got only a research area and a request for a novel idea. An expert reviewed 30 pairs, weighted toward cases the strongest judge had missed, and picked the human idea every time. The expert judged none of the generated ideas novel. Even so, the study says many of the main findings do not depend on those labels at all; if two reasonable evaluation setups disagree on the same pair, that alone shows instability.

The paper also found that more search and more reasoning were not a cure. GPT-5.6 Sol, given extra-high reasoning and live iterative search limited to arXiv, scored 0.78 macro-F1, behind Claude Opus 4.6 at 0.89, and cost roughly 10 times more per decision. A simple Claude Opus 4.6 prompt with low reasoning and majority voting across three calls did better than two dedicated novelty systems as well: 0.854 macro-F1 at $2.59, versus 0.642 at $30.66 for Idea Novelty Checker and 0.772 at $82.05 for Sakana AI’s AI Scientist.

AI2’s blunt message is that novelty evaluation is brittle, especially on AI-generated ideas, which is exactly where these judges are being used more often. The release is less a victory lap than a warning label, and that may be the most useful thing it can do.