Ai2 study finds LLM novelty judgments can shift sharply with prompt changes

Ai2 study finds LLM novelty judgments can shift sharply with prompt changes

Ai2 reports that small changes to evaluation prompts can lead large language models to give very different judgments about whether research ideas are novel. In one test, GPT-5.4’s accuracy on pairs of human-authored and AI-generated ideas fell from 93.5% to 40.9% after the prompt was simplified, and its verdict changed on more than half of the same pairs. The study also found that prompt changes affected models differently, while added search and reasoning did not consistently improve results. To test novelty judges, Ai2 built a benchmark from peer reviews of ICLR 2026 papers and paired selected human-authored ideas with ideas from a simple language-model generator. The authors caution that the generated ideas had weak labels: they assumed those ideas were less novel, rather than verifying every one individually. Escape Meridian’s reading is that the benchmark’s relevance to other scientific fields or stronger idea-generation systems remains untested. The authors say the benchmark, data, code, and an automated pipeline are available for further testing. Their findings identify a challenge for systems that use language models to assess research novelty; they do not establish whether any particular idea is genuinely new.

Publication: 660b62aa-62fb-4a9a-8979-f36c91083547

attribution — pass: Findings are consistently attributed to Ai2 and its study, with 'Ai2 reports' and 'The authors caution.' The interpretive gap is clearly labeled as Escape Meridian's reading.

Today we’re releasing Old Ideas, Novel Problems , a new study of how reliably large language models can evaluate the novelty of scientific research ideas.

summary — uncertain: Two fidelity gaps. (1) The weak-label caveat is stressed, but Ai2's mitigations are erased: an expert check that preferred the human idea in all 30 pairs, and the claim that many core findings do not depend on the labels. Smallest repair: add one sentence on each. (2) The headline finding is omitted: purpose-built evaluators scored worse than a cheap prompted Claude Opus 4.6 baseline. Also, ICLR is unexplained for general readers; a brief gloss such as 'a major machine-learning conference' would fix this.

Importantly, many of our core findings don’t depend on the labels at all. If two reasonable evaluation setups give opposite answers for the same pair of ideas, that disagreement itself reveals instability.

date — pass: originalPublished of 2026-10-27 matches the dateline on the publisher page.

Can we trust LLMs to judge whether a research idea is novel? October 27, 2026

support — pass: The 93.5% to 40.9% drop, the verdict changes, the differing model effects, the search and reasoning result, and the weak labels are all source-stated. The interpretation about untested fields and stronger generators is labeled and plausible, since the generator had no tools or literature search. The source did vary generator models, though, so 'more capable' should mean more elaborate pipelines, not stronger models.

The generator received only a research area and an instruction to propose a novel idea, with no literature search, tools, iteration, or feedback.

media — pass: The selected media is the verified direct link to the original Ai2 article, which is the most direct access to the source.