· Oct 27, 2026
Ai2 study finds LLM novelty judgments can shift sharply with prompt changes
Markdown · JSON · Original source
Ai2Comms says its study, “Old Ideas, Novel Problems,” tests how reliably can judge whether scientific research ideas are new. The team released a , data, code, and a pipeline for building evaluation sets. In tests of six LLM judges, small prompt changes sometimes altered judgments of the same ideas. For GPT-5.4 comparing human-authored and AI-generated ideas, accuracy fell from 93.5% to 40.9% after a prompt change; the model also changed its verdict on more than half of the pairs. The article says more search and reasoning did not consistently improve judgments, and two purpose-built evaluators performed worse than a simpler prompted baseline in the reported comparison. The benchmark draws on ICLR 2026 peer reviews that explicitly discussed originality. Its Human+Generated set pairs human-authored ideas with AI-generated ideas, which the authors treated as less novel using weak labels, not verified judgments for each idea. Escape Meridian’s interpretation is that the results suggest novelty evaluation may be especially unstable on AI-generated ideas; the study does not establish how well the results generalize beyond its benchmark. How will novelty judges perform on AI-generated research ideas evaluated against a broader and evolving body of scientific work?
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
· Article
Source details
Original source date: Oct 27, 2026 (2026-10-27T16:12:42.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-04T15:02:29.812Z. First observed: 2026-10-27T16:13:00.024Z. Retrieved: 2026-10-29T17:16:59.441Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The Human+Generated set uses weak labels for AI-generated ideas; the authors say these are not verified judgments for every idea. The evidence comes from the and experiments described in the article; it does not establish how results generalize to other scientific communities or future evaluation settings. The authors identify these as areas for future work.
Open question
How will novelty judges perform on AI-generated research ideas evaluated against a broader and evolving body of scientific work?
Source-stated
Ai2Comms says its study tests how reliably large language models can judge the novelty of scientific research ideas, and releases a benchmark and related resources.
Today we’re releasing Old Ideas, Novel Problems, a new study of how reliably large language models can evaluate the novelty of scientific research ideas.
Source-stated
In the GPT-5.4 comparison of human-authored and AI-generated ideas, changing the prompt lowered accuracy from 93.5% to 40.9%, and changed verdicts on more than half of the same pairs.
When we switched to the simpler prompt, accuracy dropped from 93.5% to 40.9% – below chance – and the model changed its verdict on more than half of the same pairs.
Source-stated
The authors say retrieval and additional reasoning did not consistently improve novelty judgments, and two purpose-built evaluators did worse than a simple prompted baseline in their reported comparison.
Retrieval and additional reasoning do not consistently improve judgments, and two purpose-built novelty evaluation systems perform worse than a simple prompted Claude Opus 4.6 baseline.
Source-stated
The Human+Generated set pairs human ideas with generated ideas treated as less novel using weak labels, not verified judgments for every generated idea.
We assumed the generated ideas were less novel than their human-authored counterparts; these are weak labels, not verified judgments for every idea.
Escape Meridian interpretation
The study suggests novelty evaluation may be particularly unstable on AI-generated ideas; how broadly that finding applies remains open.
Taken together, these results suggest that novelty evaluation is particularly brittle on the AI-generated ideas these systems are increasingly being asked to judge.
Source and contributor context
Hugging Face · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Ai2
Ai2 develops AI research tools and models, including the openly documented Olmo family.
Hugging Face
Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Editorial provenance
Version 1. Publication ID: fc4601f7-3d38-4cda-877a-5a921cd6ef7c.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-04T15:02:29.812Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 774966774f8a486e98375bc71ac6a939973a4ebfe74829c3923da3bfdfa97998. Source text hash: 3f7eb72d2bffeb10377060b0a228e6aa66916e1f664388682986308783af2698.
Revision and source-check history
Version 1 · 2026-11-04T15:02:29.812Z
Ai2 study finds LLM novelty judgments can shift sharply with prompt changes
Ai2Comms says its study, “Old Ideas, Novel Problems,” tests how reliably can judge whether scientific research ideas are new. The team released a , data, code, and a pipeline for building evaluation sets. In tests of six LLM judges, small prompt changes sometimes altered judgments of the same ideas. For GPT-5.4 comparing human-authored and AI-generated ideas, accuracy fell from 93.5% to 40.9% after a prompt change; the model also changed its verdict on more than half of the pairs. The article says more search and reasoning did not consistently improve judgments, and two purpose-built evaluators performed worse than a simpler prompted baseline in the reported comparison. The benchmark draws on ICLR 2026 peer reviews that explicitly discussed originality. Its Human+Generated set pairs human-authored ideas with AI-generated ideas, which the authors treated as less novel using weak labels, not verified judgments for each idea. Escape Meridian’s interpretation is that the results suggest novelty evaluation may be especially unstable on AI-generated ideas; the study does not establish how well the results generalize beyond its benchmark. How will novelty judges perform on AI-generated research ideas evaluated against a broader and evolving body of scientific work?
Original source
· Article
Source details
Original source date: Oct 27, 2026 (2026-10-27T16:12:42.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-04T15:02:29.812Z. First observed: 2026-10-27T16:13:00.024Z. Retrieved: 2026-10-29T17:16:59.441Z.
Limits
The Human+Generated set uses weak labels for AI-generated ideas; the authors say these are not verified judgments for every idea. The evidence comes from the and experiments described in the article; it does not establish how results generalize to other scientific communities or future evaluation settings. The authors identify these as areas for future work.
Open question
How will novelty judges perform on AI-generated research ideas evaluated against a broader and evolving body of scientific work?
Source-stated
Ai2Comms says its study tests how reliably large language models can judge the novelty of scientific research ideas, and releases a benchmark and related resources.
Today we’re releasing Old Ideas, Novel Problems, a new study of how reliably large language models can evaluate the novelty of scientific research ideas.
Source-stated
In the GPT-5.4 comparison of human-authored and AI-generated ideas, changing the prompt lowered accuracy from 93.5% to 40.9%, and changed verdicts on more than half of the same pairs.
When we switched to the simpler prompt, accuracy dropped from 93.5% to 40.9% – below chance – and the model changed its verdict on more than half of the same pairs.
Source-stated
The authors say retrieval and additional reasoning did not consistently improve novelty judgments, and two purpose-built evaluators did worse than a simple prompted baseline in their reported comparison.
Retrieval and additional reasoning do not consistently improve judgments, and two purpose-built novelty evaluation systems perform worse than a simple prompted Claude Opus 4.6 baseline.
Source-stated
The Human+Generated set pairs human ideas with generated ideas treated as less novel using weak labels, not verified judgments for every generated idea.
We assumed the generated ideas were less novel than their human-authored counterparts; these are weak labels, not verified judgments for every idea.
Escape Meridian interpretation
The study suggests novelty evaluation may be particularly unstable on AI-generated ideas; how broadly that finding applies remains open.
Taken together, these results suggest that novelty evaluation is particularly brittle on the AI-generated ideas these systems are increasingly being asked to judge.
Source and contributor context
Hugging Face · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Ai2
Ai2 develops AI research tools and models, including the openly documented Olmo family.
Hugging Face
Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Editorial provenance
Version 1. Publication ID: fc4601f7-3d38-4cda-877a-5a921cd6ef7c.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-04T15:02:29.812Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 774966774f8a486e98375bc71ac6a939973a4ebfe74829c3923da3bfdfa97998. Source text hash: 3f7eb72d2bffeb10377060b0a228e6aa66916e1f664388682986308783af2698.
uncertain · 2026-11-04T15:02:29.812Z · anthropic/claude-opus-5.5
Publication: fc4601f7-3d38-4cda-877a-5a921cd6ef7c
attribution — uncertain: The draft labels the conclusion that novelty evaluation may be especially unstable on AI-generated ideas as 'Escape Meridian's interpretation.' The authors state this conclusion themselves, so the draft assigns the source's own thesis to EM. Smallest repair: attribute it to the authors, for example 'The authors conclude...'. Separately, 'Ai2Comms says' is acceptable but loose: the byline lists Kyle Wiggers, Ai2Comms and allenai, and the post speaks for Ai2.
Taken together, these results suggest that novelty evaluation is particularly brittle on the AI-generated ideas these systems are increasingly being asked to judge.
summary — pass: The summary is understandable on its own. It names the actor and the study, explains that the judges assess whether research ideas are new, and gives the headline prompt-sensitivity result, the search and reasoning finding, and the purpose-built evaluator finding. It also notes the weak-label caveat. The question follows from the authors' call for evolving benchmarks. One small gap that does not block understanding: the 93.5% prompt told the model that exactly one idea per pair was novel.
One prompt told the model that reviewers had found exactly one idea in each pair to be novel and asked it to identify which one; the other simply asked which idea was novel.
date — pass: The originalPublished value of 2026-10-27 matches the article's stated publication date of October 27, 2026.
Published October 27, 2026
support — pass: Each claim matches the source text: the release, the drop from 93.5% to 40.9% with changed verdicts on more than half of pairs, the finding that retrieval and reasoning did not consistently help, the purpose-built evaluators underperforming the baseline, and the weak labels. The limitations stay within the source and do not imply unsupported benefits or costs.
We assumed the generated ideas were less novel than their human-authored counterparts; these are weak labels, not verified judgments for every idea.
media — pass: The selected media is the verified link to the original Hugging Face article, which is the direct source of this reading.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Explore Hugging FaceA language model with many learned numerical settings, used to process and generate language.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceHugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Explore Hugging FaceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceHugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Explore Hugging FaceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference