# Ai2 study finds LLM novelty judgments can shift sharply with prompt changes [Canonical reading](https://escapemeridian.com/readings/ai2-study-finds-llm-novelty-judgments-can-shift-sharply-with-ffdef57b) Ai2 reports that small changes to evaluation prompts can lead large language models to give very different judgments about whether research ideas are novel. In one test, GPT-5.4’s accuracy on pairs of human-authored and AI-generated ideas fell from 93.5% to 40.9% after the prompt was simplified, and its verdict changed on more than half of the same pairs. The study also found that prompt changes affected models differently, while added search and reasoning did not consistently improve results. To test novelty judges, Ai2 built a benchmark from peer reviews of ICLR 2026 papers and paired selected human-authored ideas with ideas from a simple language-model generator. The authors caution that the generated ideas had weak labels: they assumed those ideas were less novel, rather than verifying every one individually. Escape Meridian’s reading is that the benchmark’s relevance to other scientific fields or stronger idea-generation systems remains untested. The authors say the benchmark, data, code, and an automated pipeline are available for further testing. Their findings identify a challenge for systems that use language models to assess research novelty; they do not establish whether any particular idea is genuinely new. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source Ai2 · Article Original source date: Oct 27, 2026 (2026-10-27T08:00:00.000Z). Date basis: Date supplied by the publisher’s feed. Added to Escape Meridian: 2026-11-05T08:47:16.704Z. First observed: 2026-10-27T16:17:11.824Z. Retrieved: 2026-10-29T15:47:15.356Z. - [Can we trust LLMs to judge whether a research idea is novel? | Ai2](https://allenai.org/blog/novelty-bench) ## Limits The generated ideas in the Human+Generated benchmark were assigned weak labels; Ai2 did not verify the novelty of every generated idea. The evidence describes tests on selected ICLR 2026 research ideas and a simple LLM generator; it does not establish performance across other scientific fields or more capable idea-generation systems. The source is a publisher account of the study; the reading does not independently assess its methods or results. ## Open question How well will novelty judges perform on AI-generated ideas from other scientific fields and more capable idea-generation systems? ## Source-stated Ai2 reports that changing a prompt can sharply alter a model’s accuracy and verdicts when judging the novelty of human-authored and AI-generated ideas. > When we switched to the simpler prompt, accuracy dropped from 93.5% to 40.9% – below chance – and the model changed its verdict on more than half of the same pairs. - [Supporting source](https://allenai.org/blog/novelty-bench) ## Source-stated The study found that added search and reasoning did not consistently improve novelty judgments. > More search and reasoning didn’t necessarily produce better novelty judgments. - [Supporting source](https://allenai.org/blog/novelty-bench) ## Source-stated Ai2’s Human+Generated benchmark pairs selected human-authored ideas with generated ideas that were assumed, but not individually verified, to be less novel. > We assumed the generated ideas were less novel than their human-authored counterparts; these are weak labels, not verified judgments for every idea. - [Supporting source](https://allenai.org/blog/novelty-bench) ## Escape Meridian interpretation The study’s findings leave open how well the benchmark applies to other scientific fields and future generations of novelty judges. > researchers can apply them to new peer reviews, other scientific communities, and future generations of novelty judges. - [Supporting source](https://allenai.org/blog/novelty-bench) ## Source and contributor context Ai2 · Publisher The publisher is identified here. No individual author has been inferred. ## Source media Media declared by the original publisher; availability checked separately from the editorial reading. - [A light bulb at the center of six AI model icons, some marked with check marks and others with crosses. · Ai2](https://www.datocms-assets.com/64837/1793023853-novelty-bench-header.png) ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Ai2 Ai2 develops AI research tools and models, including the openly documented Olmo family. - [Explore this context](/entities/ai2) - [Reference](https://allenai.org/) - [Reference](https://allenai.org/olmo) ## Editorial provenance Version 1. Publication ID: 660b62aa-62fb-4a9a-8979-f36c91083547. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-05T08:47:16.704Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 39a94865967c791531f756cb4d440c43b4659983ed2172ccc08b4d83faa8052d. Source text hash: 323a8512a914785f86f089493fdf49a8a17b368e72c3402497f0be682b6da18f. ## Revision and source-check history ### Version 1 — 2026-11-05T08:47:16.704Z Ai2 reports that small changes to evaluation prompts can lead large language models to give very different judgments about whether research ideas are novel. In one test, GPT-5.4’s accuracy on pairs of human-authored and AI-generated ideas fell from 93.5% to 40.9% after the prompt was simplified, and its verdict changed on more than half of the same pairs. The study also found that prompt changes affected models differently, while added search and reasoning did not consistently improve results. To test novelty judges, Ai2 built a benchmark from peer reviews of ICLR 2026 papers and paired selected human-authored ideas with ideas from a simple language-model generator. The authors caution that the generated ideas had weak labels: they assumed those ideas were less novel, rather than verifying every one individually. Escape Meridian’s reading is that the benchmark’s relevance to other scientific fields or stronger idea-generation systems remains untested. The authors say the benchmark, data, code, and an automated pipeline are available for further testing. Their findings identify a challenge for systems that use language models to assess research novelty; they do not establish whether any particular idea is genuinely new. ## Original source Ai2 · Article Original source date: Oct 27, 2026 (2026-10-27T08:00:00.000Z). Date basis: Date supplied by the publisher’s feed. Added to Escape Meridian: 2026-11-05T08:47:16.704Z. First observed: 2026-10-27T16:17:11.824Z. Retrieved: 2026-10-29T15:47:15.356Z. - [Can we trust LLMs to judge whether a research idea is novel? | Ai2](https://allenai.org/blog/novelty-bench) ## Limits The generated ideas in the Human+Generated benchmark were assigned weak labels; Ai2 did not verify the novelty of every generated idea. The evidence describes tests on selected ICLR 2026 research ideas and a simple LLM generator; it does not establish performance across other scientific fields or more capable idea-generation systems. The source is a publisher account of the study; the reading does not independently assess its methods or results. ## Open question How well will novelty judges perform on AI-generated ideas from other scientific fields and more capable idea-generation systems? ## Source-stated Ai2 reports that changing a prompt can sharply alter a model’s accuracy and verdicts when judging the novelty of human-authored and AI-generated ideas. > When we switched to the simpler prompt, accuracy dropped from 93.5% to 40.9% – below chance – and the model changed its verdict on more than half of the same pairs. - [Supporting source](https://allenai.org/blog/novelty-bench) ## Source-stated The study found that added search and reasoning did not consistently improve novelty judgments. > More search and reasoning didn’t necessarily produce better novelty judgments. - [Supporting source](https://allenai.org/blog/novelty-bench) ## Source-stated Ai2’s Human+Generated benchmark pairs selected human-authored ideas with generated ideas that were assumed, but not individually verified, to be less novel. > We assumed the generated ideas were less novel than their human-authored counterparts; these are weak labels, not verified judgments for every idea. - [Supporting source](https://allenai.org/blog/novelty-bench) ## Escape Meridian interpretation The study’s findings leave open how well the benchmark applies to other scientific fields and future generations of novelty judges. > researchers can apply them to new peer reviews, other scientific communities, and future generations of novelty judges. - [Supporting source](https://allenai.org/blog/novelty-bench) ## Source and contributor context Ai2 · Publisher The publisher is identified here. No individual author has been inferred. ## Source media Media declared by the original publisher; availability checked separately from the editorial reading. - [A light bulb at the center of six AI model icons, some marked with check marks and others with crosses. · Ai2](https://www.datocms-assets.com/64837/1793023853-novelty-bench-header.png) ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Ai2 Ai2 develops AI research tools and models, including the openly documented Olmo family. - [Explore this context](/entities/ai2) - [Reference](https://allenai.org/) - [Reference](https://allenai.org/olmo) ## Editorial provenance Version 1. Publication ID: 660b62aa-62fb-4a9a-8979-f36c91083547. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-05T08:47:16.704Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 39a94865967c791531f756cb4d440c43b4659983ed2172ccc08b4d83faa8052d. Source text hash: 323a8512a914785f86f089493fdf49a8a17b368e72c3402497f0be682b6da18f. ### uncertain — 2026-11-05T08:47:16.704Z — anthropic/claude-opus-5.5 Publication: 660b62aa-62fb-4a9a-8979-f36c91083547 attribution — pass: Findings are consistently attributed to Ai2 and its study, with 'Ai2 reports' and 'The authors caution.' The interpretive gap is clearly labeled as Escape Meridian's reading. > Today we’re releasing Old Ideas, Novel Problems , a new study of how reliably large language models can evaluate the novelty of scientific research ideas. summary — uncertain: Two fidelity gaps. (1) The weak-label caveat is stressed, but Ai2's mitigations are erased: an expert check that preferred the human idea in all 30 pairs, and the claim that many core findings do not depend on the labels. Smallest repair: add one sentence on each. (2) The headline finding is omitted: purpose-built evaluators scored worse than a cheap prompted Claude Opus 4.6 baseline. Also, ICLR is unexplained for general readers; a brief gloss such as 'a major machine-learning conference' would fix this. > Importantly, many of our core findings don’t depend on the labels at all. If two reasonable evaluation setups give opposite answers for the same pair of ideas, that disagreement itself reveals instability. date — pass: originalPublished of 2026-10-27 matches the dateline on the publisher page. > Can we trust LLMs to judge whether a research idea is novel? October 27, 2026 support — pass: The 93.5% to 40.9% drop, the verdict changes, the differing model effects, the search and reasoning result, and the weak labels are all source-stated. The interpretation about untested fields and stronger generators is labeled and plausible, since the generator had no tools or literature search. The source did vary generator models, though, so 'more capable' should mean more elaborate pipelines, not stronger models. > The generator received only a research area and an instruction to propose a novel idea, with no literature search, tools, iteration, or feedback. media — pass: The selected media is the verified direct link to the original Ai2 article, which is the most direct access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/ffdef57bfef4a5c1ef423d41c052ad57fb0a17c67eb09bab6dc9b838c0fef33a/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/ai2-study-finds-llm-novelty-judgments-can-shift-sharply-with-ffdef57b.json)