Can we trust LLMs to judge whether a research idea is novel?
Hugging Face ● Covered by 2 sources
Large language model novelty judges were tested for how reliably they decide whether research ideas are genuinely novel relative to prior work, and they produced inconsistent verdicts for the same ideas when evaluation prompts or setups changed. Accuracy for GPT-5.4 dropped from 93.5% to 40.9% after switching to a simpler prompt on Human+Generated pairs. The findings change how such judges should be used, suggesting novelty evaluation is brittle—especially on AI-generated ideas—and motivating new benchmarks and tools to measure and improve stability.