AI2 released a study benchmarking how reliably large language models can judge the novelty of research ideas, finding results that vary sharply with prompting and evaluation setup
Benchmark result Provisional 86% confidence first seen
Allen Institute for AI (AI2) published a study testing LLM-based “novelty judges” for whether they can reliably determine if research ideas are genuinely novel. The benchmark found that accuracy can drop dramatically when prompts or setups are changed, indicating novelty evaluation is brittle and sensitive to evaluation design.