{"id":"569d4c53221aa0b8a1d3c9d9f941d4cdb544bc661e5622e1d33098e554cebba9","title":"Ai2 study finds LLM novelty judgments can shift sharply with prompt changes","argument":"Ai2Comms says its study, “Old Ideas, Novel Problems,” tests how reliably large language models can judge whether scientific research ideas are new. The team released a benchmark, data, code, and a pipeline for building evaluation sets.\n\nIn tests of six LLM judges, small prompt changes sometimes altered judgments of the same ideas. For GPT-5.4 comparing human-authored and AI-generated ideas, accuracy fell from 93.5% to 40.9% after a prompt change; the model also changed its verdict on more than half of the pairs. The article says more search and reasoning did not consistently improve judgments, and two purpose-built evaluators performed worse than a simpler prompted baseline in the reported comparison.\n\nThe benchmark draws on ICLR 2026 peer reviews that explicitly discussed originality. Its Human+Generated set pairs human-authored ideas with AI-generated ideas, which the authors treated as less novel using weak labels, not verified judgments for each idea. Escape Meridian’s interpretation is that the results suggest novelty evaluation may be especially unstable on AI-generated ideas; the study does not establish how well the results generalize beyond its benchmark. How will novelty judges perform on AI-generated research ideas evaluated against a broader and evolving body of scientific work?","topicIds":["intelligence-and-discovery","work-and-agency"],"domain":"work","claimType":"attributed reading","limits":"The Human+Generated set uses weak labels for AI-generated ideas; the authors say these are not verified judgments for every idea. The evidence comes from the benchmark and experiments described in the article; it does not establish how results generalize to other scientific communities or future evaluation settings. The authors identify these as areas for future work.","speaker":{"name":"Hugging Face","roleAtSource":"Publisher"},"source":{"title":"Can we trust LLMs to judge whether a research idea is novel?","publisher":"Hugging Face","format":"Article","url":"https://huggingface.co/blog/allenai/novelty-bench","publishedAt":"2026-10-27T16:12:42.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"Ai2Comms says its study tests how reliably large language models can judge the novelty of scientific research ideas, and releases a benchmark and related resources.","excerpt":"Today we’re releasing Old Ideas, Novel Problems, a new study of how reliably large language models can evaluate the novelty of scientific research ideas.","url":"https://huggingface.co/blog/allenai/novelty-bench","relationship":"Source-stated"},{"text":"In the GPT-5.4 comparison of human-authored and AI-generated ideas, changing the prompt lowered accuracy from 93.5% to 40.9%, and changed verdicts on more than half of the same pairs.","excerpt":"When we switched to the simpler prompt, accuracy dropped from 93.5% to 40.9% – below chance – and the model changed its verdict on more than half of the same pairs.","url":"https://huggingface.co/blog/allenai/novelty-bench","relationship":"Source-stated"},{"text":"The authors say retrieval and additional reasoning did not consistently improve novelty judgments, and two purpose-built evaluators did worse than a simple prompted baseline in their reported comparison.","excerpt":"Retrieval and additional reasoning do not consistently improve judgments, and two purpose-built novelty evaluation systems perform worse than a simple prompted Claude Opus 4.6 baseline.","url":"https://huggingface.co/blog/allenai/novelty-bench","relationship":"Source-stated"},{"text":"The Human+Generated set pairs human ideas with generated ideas treated as less novel using weak labels, not verified judgments for every generated idea.","excerpt":"We assumed the generated ideas were less novel than their human-authored counterparts; these are weak labels, not verified judgments for every idea.","url":"https://huggingface.co/blog/allenai/novelty-bench","relationship":"Source-stated"},{"text":"The study suggests novelty evaluation may be particularly unstable on AI-generated ideas; how broadly that finding applies remains open.","excerpt":"Taken together, these results suggest that novelty evaluation is particularly brittle on the AI-generated ideas these systems are increasingly being asked to judge.","url":"https://huggingface.co/blog/allenai/novelty-bench","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"found","sourceUrl":"https://huggingface.co/blog/allenai/novelty-bench","checkedAt":"2026-11-05T08:06:07.523Z","items":[],"note":"Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How will novelty judges perform on AI-generated research ideas evaluated against a broader and evolving body of scientific work?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Ai2 study finds LLM novelty judgments can shift sharply with prompt changes","summary":"Ai2Comms says its study, “Old Ideas, Novel Problems,” tests how reliably large language models can judge whether scientific research ideas are new. The team released a benchmark, data, code, and a pipeline for building evaluation sets.\n\nIn tests of six LLM judges, small prompt changes sometimes altered judgments of the same ideas. For GPT-5.4 comparing human-authored and AI-generated ideas, accuracy fell from 93.5% to 40.9% after a prompt change; the model also changed its verdict on more than half of the pairs. The article says more search and reasoning did not consistently improve judgments, and two purpose-built evaluators performed worse than a simpler prompted baseline in the reported comparison.\n\nThe benchmark draws on ICLR 2026 peer reviews that explicitly discussed originality. Its Human+Generated set pairs human-authored ideas with AI-generated ideas, which the authors treated as less novel using weak labels, not verified judgments for each idea. Escape Meridian’s interpretation is that the results suggest novelty evaluation may be especially unstable on AI-generated ideas; the study does not establish how well the results generalize beyond its benchmark. How will novelty judges perform on AI-generated research ideas evaluated against a broader and evolving body of scientific work?","question":"How will novelty judges perform on AI-generated research ideas evaluated against a broader and evolving body of scientific work?","claims":[{"text":"Ai2Comms says its study tests how reliably large language models can judge the novelty of scientific research ideas, and releases a benchmark and related resources.","excerpt":"Today we’re releasing Old Ideas, Novel Problems, a new study of how reliably large language models can evaluate the novelty of scientific research ideas.","support":"source-stated"},{"text":"In the GPT-5.4 comparison of human-authored and AI-generated ideas, changing the prompt lowered accuracy from 93.5% to 40.9%, and changed verdicts on more than half of the same pairs.","excerpt":"When we switched to the simpler prompt, accuracy dropped from 93.5% to 40.9% – below chance – and the model changed its verdict on more than half of the same pairs.","support":"source-stated"},{"text":"The authors say retrieval and additional reasoning did not consistently improve novelty judgments, and two purpose-built evaluators did worse than a simple prompted baseline in their reported comparison.","excerpt":"Retrieval and additional reasoning do not consistently improve judgments, and two purpose-built novelty evaluation systems perform worse than a simple prompted Claude Opus 4.6 baseline.","support":"source-stated"},{"text":"The Human+Generated set pairs human ideas with generated ideas treated as less novel using weak labels, not verified judgments for every generated idea.","excerpt":"We assumed the generated ideas were less novel than their human-authored counterparts; these are weak labels, not verified judgments for every idea.","support":"source-stated"},{"text":"The study suggests novelty evaluation may be particularly unstable on AI-generated ideas; how broadly that finding applies remains open.","excerpt":"Taken together, these results suggest that novelty evaluation is particularly brittle on the AI-generated ideas these systems are increasingly being asked to judge.","support":"interpretation"}],"limitations":["The Human+Generated set uses weak labels for AI-generated ideas; the authors say these are not verified judgments for every idea.","The evidence comes from the benchmark and experiments described in the article; it does not establish how results generalize to other scientific communities or future evaluation settings. The authors identify these as areas for future work."],"mediaId":"1d6c5c59e46e52b936cbc7b983785a69d62ade05b868a690011ee911fd57d509","correctionReason":null,"id":"569d4c53221aa0b8a1d3c9d9f941d4cdb544bc661e5622e1d33098e554cebba9","publicationId":"fc4601f7-3d38-4cda-877a-5a921cd6ef7c","version":1,"sourceUrl":"https://huggingface.co/blog/allenai/novelty-bench","sourceTitle":"Can we trust LLMs to judge whether a research idea is novel?","publisher":"Hugging Face","publishedAt":"2026-10-27T16:12:42.000Z","originalPublished":{"value":"2026-10-27T16:12:42.000Z","precision":"instant","basis":"feed_entry"},"firstObservedAt":"2026-10-27T16:13:00.024Z","createdAt":"2026-11-04T15:02:29.812Z","historical":true,"entityIds":["hugging-face"],"topicIds":["intelligence-and-discovery","work-and-agency"],"media":[{"id":"1d6c5c59e46e52b936cbc7b983785a69d62ade05b868a690011ee911fd57d509","kind":"article","url":"https://huggingface.co/blog/allenai/novelty-bench","sourceUrl":"https://huggingface.co/blog/allenai/novelty-bench","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-29T17:16:59.622Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"found","sourceUrl":"https://huggingface.co/blog/allenai/novelty-bench","checkedAt":"2026-11-05T08:06:07.523Z","items":[],"note":"Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"1d6c5c59e46e52b936cbc7b983785a69d62ade05b868a690011ee911fd57d509","confidence":0.95,"provider":"typesafe","model":"jev-1.13.0","inputHash":"ffdf363e6e595aebbe8780679341f03340b62bf7fa7b0df1a2b5f333ed294d78","decidedAt":"2026-10-29T17:17:00.464Z"},"provenance":{"textHash":"3f7eb72d2bffeb10377060b0a228e6aa66916e1f664388682986308783af2698","observationId":"774966774f8a486e98375bc71ac6a939973a4ebfe74829c3923da3bfdfa97998","retrievedAt":"2026-10-29T17:16:59.441Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"ef234449-edb2-40ce-86e9-b52dcc2ec570"},"reviewer":{"model":"gpt-6-luna","runId":"24bac3c1-56a4-423a-87e1-ea7728974153"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-04T15:02:29.812Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[],"authorRole":"article-author","hosts":[],"status":"missing","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[],"participationCoverage":{"total":0,"returned":0,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://huggingface.co/blog/allenai/novelty-bench","snapshotId":"565a1bff626af9d832f330f15b2e1355a6302036849690f0b26e076a45e6067c","contentSha256":"f96a53fe4d2792f6dda9afa36be88e9845dcb1f55a25520ee9dba9c95edfc264","textHash":"3f7eb72d2bffeb10377060b0a228e6aa66916e1f664388682986308783af2698","observedAt":"2026-10-29T17:16:59.441Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/569d4c53221aa0b8a1d3c9d9f941d4cdb544bc661e5622e1d33098e554cebba9/source"}},"canonicalUrl":"https://escapemeridian.com/readings/ai2-study-finds-llm-novelty-judgments-can-shift-sharply-with-569d4c53","formats":{"html":"/readings/ai2-study-finds-llm-novelty-judgments-can-shift-sharply-with-569d4c53","markdown":"/readings/ai2-study-finds-llm-novelty-judgments-can-shift-sharply-with-569d4c53.md","json":"/readings/ai2-study-finds-llm-novelty-judgments-can-shift-sharply-with-569d4c53.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"},{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"}],"entities":[{"id":"ai2","name":"Ai2","kind":"organization","context":"Ai2 develops AI research tools and models, including the openly documented Olmo family.","referenceUrls":["https://allenai.org/","https://allenai.org/olmo"],"path":"/entities/ai2","json":"/entities/ai2.json","markdown":"/entities/ai2.md"},{"id":"hugging-face","name":"Hugging Face","kind":"organization","context":"Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.","referenceUrls":["https://huggingface.co/blog"],"path":"/entities/hugging-face","json":"/entities/hugging-face.json","markdown":"/entities/hugging-face.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"569d4c53221aa0b8a1d3c9d9f941d4cdb544bc661e5622e1d33098e554cebba9","revision":"fc4601f7-3d38-4cda-877a-5a921cd6ef7c","revisionKind":"publication-id","title":"Ai2 study finds LLM novelty judgments can shift sharply with prompt changes","url":"https://escapemeridian.com/readings/ai2-study-finds-llm-novelty-judgments-can-shift-sharply-with-569d4c53"},"source":{"type":"source","id":"https://huggingface.co/blog/allenai/novelty-bench","revision":"3f7eb72d2bffeb10377060b0a228e6aa66916e1f664388682986308783af2698","revisionKind":"sha256-source-text","title":"Can we trust LLMs to judge whether a research idea is novel?","url":"https://huggingface.co/blog/allenai/novelty-bench"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"569d4c53221aa0b8a1d3c9d9f941d4cdb544bc661e5622e1d33098e554cebba9","title":"Ai2 study finds LLM novelty judgments can shift sharply with prompt changes","argument":"Ai2Comms says its study, “Old Ideas, Novel Problems,” tests how reliably large language models can judge whether scientific research ideas are new. The team released a benchmark, data, code, and a pipeline for building evaluation sets.\n\nIn tests of six LLM judges, small prompt changes sometimes altered judgments of the same ideas. For GPT-5.4 comparing human-authored and AI-generated ideas, accuracy fell from 93.5% to 40.9% after a prompt change; the model also changed its verdict on more than half of the pairs. The article says more search and reasoning did not consistently improve judgments, and two purpose-built evaluators performed worse than a simpler prompted baseline in the reported comparison.\n\nThe benchmark draws on ICLR 2026 peer reviews that explicitly discussed originality. Its Human+Generated set pairs human-authored ideas with AI-generated ideas, which the authors treated as less novel using weak labels, not verified judgments for each idea. Escape Meridian’s interpretation is that the results suggest novelty evaluation may be especially unstable on AI-generated ideas; the study does not establish how well the results generalize beyond its benchmark. How will novelty judges perform on AI-generated research ideas evaluated against a broader and evolving body of scientific work?","topicIds":["intelligence-and-discovery","work-and-agency"],"domain":"work","claimType":"attributed reading","limits":"The Human+Generated set uses weak labels for AI-generated ideas; the authors say these are not verified judgments for every idea. The evidence comes from the benchmark and experiments described in the article; it does not establish how results generalize to other scientific communities or future evaluation settings. The authors identify these as areas for future work.","speaker":{"name":"Hugging Face","roleAtSource":"Publisher"},"source":{"title":"Can we trust LLMs to judge whether a research idea is novel?","publisher":"Hugging Face","format":"Article","url":"https://huggingface.co/blog/allenai/novelty-bench","publishedAt":"2026-10-27T16:12:42.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"Ai2Comms says its study tests how reliably large language models can judge the novelty of scientific research ideas, and releases a benchmark and related resources.","excerpt":"Today we’re releasing Old Ideas, Novel Problems, a new study of how reliably large language models can evaluate the novelty of scientific research ideas.","url":"https://huggingface.co/blog/allenai/novelty-bench","relationship":"Source-stated"},{"text":"In the GPT-5.4 comparison of human-authored and AI-generated ideas, changing the prompt lowered accuracy from 93.5% to 40.9%, and changed verdicts on more than half of the same pairs.","excerpt":"When we switched to the simpler prompt, accuracy dropped from 93.5% to 40.9% – below chance – and the model changed its verdict on more than half of the same pairs.","url":"https://huggingface.co/blog/allenai/novelty-bench","relationship":"Source-stated"},{"text":"The authors say retrieval and additional reasoning did not consistently improve novelty judgments, and two purpose-built evaluators did worse than a simple prompted baseline in their reported comparison.","excerpt":"Retrieval and additional reasoning do not consistently improve judgments, and two purpose-built novelty evaluation systems perform worse than a simple prompted Claude Opus 4.6 baseline.","url":"https://huggingface.co/blog/allenai/novelty-bench","relationship":"Source-stated"},{"text":"The Human+Generated set pairs human ideas with generated ideas treated as less novel using weak labels, not verified judgments for every generated idea.","excerpt":"We assumed the generated ideas were less novel than their human-authored counterparts; these are weak labels, not verified judgments for every idea.","url":"https://huggingface.co/blog/allenai/novelty-bench","relationship":"Source-stated"},{"text":"The study suggests novelty evaluation may be particularly unstable on AI-generated ideas; how broadly that finding applies remains open.","excerpt":"Taken together, these results suggest that novelty evaluation is particularly brittle on the AI-generated ideas these systems are increasingly being asked to judge.","url":"https://huggingface.co/blog/allenai/novelty-bench","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"found","sourceUrl":"https://huggingface.co/blog/allenai/novelty-bench","checkedAt":"2026-11-05T08:06:07.523Z","items":[],"note":"Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How will novelty judges perform on AI-generated research ideas evaluated against a broader and evolving body of scientific work?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Ai2 study finds LLM novelty judgments can shift sharply with prompt changes","summary":"Ai2Comms says its study, “Old Ideas, Novel Problems,” tests how reliably large language models can judge whether scientific research ideas are new. The team released a benchmark, data, code, and a pipeline for building evaluation sets.\n\nIn tests of six LLM judges, small prompt changes sometimes altered judgments of the same ideas. For GPT-5.4 comparing human-authored and AI-generated ideas, accuracy fell from 93.5% to 40.9% after a prompt change; the model also changed its verdict on more than half of the pairs. The article says more search and reasoning did not consistently improve judgments, and two purpose-built evaluators performed worse than a simpler prompted baseline in the reported comparison.\n\nThe benchmark draws on ICLR 2026 peer reviews that explicitly discussed originality. Its Human+Generated set pairs human-authored ideas with AI-generated ideas, which the authors treated as less novel using weak labels, not verified judgments for each idea. Escape Meridian’s interpretation is that the results suggest novelty evaluation may be especially unstable on AI-generated ideas; the study does not establish how well the results generalize beyond its benchmark. How will novelty judges perform on AI-generated research ideas evaluated against a broader and evolving body of scientific work?","question":"How will novelty judges perform on AI-generated research ideas evaluated against a broader and evolving body of scientific work?","claims":[{"text":"Ai2Comms says its study tests how reliably large language models can judge the novelty of scientific research ideas, and releases a benchmark and related resources.","excerpt":"Today we’re releasing Old Ideas, Novel Problems, a new study of how reliably large language models can evaluate the novelty of scientific research ideas.","support":"source-stated"},{"text":"In the GPT-5.4 comparison of human-authored and AI-generated ideas, changing the prompt lowered accuracy from 93.5% to 40.9%, and changed verdicts on more than half of the same pairs.","excerpt":"When we switched to the simpler prompt, accuracy dropped from 93.5% to 40.9% – below chance – and the model changed its verdict on more than half of the same pairs.","support":"source-stated"},{"text":"The authors say retrieval and additional reasoning did not consistently improve novelty judgments, and two purpose-built evaluators did worse than a simple prompted baseline in their reported comparison.","excerpt":"Retrieval and additional reasoning do not consistently improve judgments, and two purpose-built novelty evaluation systems perform worse than a simple prompted Claude Opus 4.6 baseline.","support":"source-stated"},{"text":"The Human+Generated set pairs human ideas with generated ideas treated as less novel using weak labels, not verified judgments for every generated idea.","excerpt":"We assumed the generated ideas were less novel than their human-authored counterparts; these are weak labels, not verified judgments for every idea.","support":"source-stated"},{"text":"The study suggests novelty evaluation may be particularly unstable on AI-generated ideas; how broadly that finding applies remains open.","excerpt":"Taken together, these results suggest that novelty evaluation is particularly brittle on the AI-generated ideas these systems are increasingly being asked to judge.","support":"interpretation"}],"limitations":["The Human+Generated set uses weak labels for AI-generated ideas; the authors say these are not verified judgments for every idea.","The evidence comes from the benchmark and experiments described in the article; it does not establish how results generalize to other scientific communities or future evaluation settings. The authors identify these as areas for future work."],"mediaId":"1d6c5c59e46e52b936cbc7b983785a69d62ade05b868a690011ee911fd57d509","correctionReason":null,"id":"569d4c53221aa0b8a1d3c9d9f941d4cdb544bc661e5622e1d33098e554cebba9","publicationId":"fc4601f7-3d38-4cda-877a-5a921cd6ef7c","version":1,"sourceUrl":"https://huggingface.co/blog/allenai/novelty-bench","sourceTitle":"Can we trust LLMs to judge whether a research idea is novel?","publisher":"Hugging Face","publishedAt":"2026-10-27T16:12:42.000Z","originalPublished":{"value":"2026-10-27T16:12:42.000Z","precision":"instant","basis":"feed_entry"},"firstObservedAt":"2026-10-27T16:13:00.024Z","createdAt":"2026-11-04T15:02:29.812Z","historical":true,"entityIds":["hugging-face"],"topicIds":["intelligence-and-discovery","work-and-agency"],"media":[{"id":"1d6c5c59e46e52b936cbc7b983785a69d62ade05b868a690011ee911fd57d509","kind":"article","url":"https://huggingface.co/blog/allenai/novelty-bench","sourceUrl":"https://huggingface.co/blog/allenai/novelty-bench","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-29T17:16:59.622Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"found","sourceUrl":"https://huggingface.co/blog/allenai/novelty-bench","checkedAt":"2026-11-05T08:06:07.523Z","items":[],"note":"Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"1d6c5c59e46e52b936cbc7b983785a69d62ade05b868a690011ee911fd57d509","confidence":0.95,"provider":"typesafe","model":"jev-1.13.0","inputHash":"ffdf363e6e595aebbe8780679341f03340b62bf7fa7b0df1a2b5f333ed294d78","decidedAt":"2026-10-29T17:17:00.464Z"},"provenance":{"textHash":"3f7eb72d2bffeb10377060b0a228e6aa66916e1f664388682986308783af2698","observationId":"774966774f8a486e98375bc71ac6a939973a4ebfe74829c3923da3bfdfa97998","retrievedAt":"2026-10-29T17:16:59.441Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"ef234449-edb2-40ce-86e9-b52dcc2ec570"},"reviewer":{"model":"gpt-6-luna","runId":"24bac3c1-56a4-423a-87e1-ea7728974153"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-04T15:02:29.812Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[],"authorRole":"article-author","hosts":[],"status":"missing","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[],"participationCoverage":{"total":0,"returned":0,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://huggingface.co/blog/allenai/novelty-bench","snapshotId":"565a1bff626af9d832f330f15b2e1355a6302036849690f0b26e076a45e6067c","contentSha256":"f96a53fe4d2792f6dda9afa36be88e9845dcb1f55a25520ee9dba9c95edfc264","textHash":"3f7eb72d2bffeb10377060b0a228e6aa66916e1f664388682986308783af2698","observedAt":"2026-10-29T17:16:59.441Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/569d4c53221aa0b8a1d3c9d9f941d4cdb544bc661e5622e1d33098e554cebba9/source"}}}],"audits":[{"publicationId":"fc4601f7-3d38-4cda-877a-5a921cd6ef7c","verdict":"uncertain","findings":[{"dimension":"attribution","status":"uncertain","explanation":"The draft labels the conclusion that novelty evaluation may be especially unstable on AI-generated ideas as 'Escape Meridian's interpretation.' The authors state this conclusion themselves, so the draft assigns the source's own thesis to EM. Smallest repair: attribute it to the authors, for example 'The authors conclude...'. Separately, 'Ai2Comms says' is acceptable but loose: the byline lists Kyle Wiggers, Ai2Comms and allenai, and the post speaks for Ai2.","excerpt":"Taken together, these results suggest that novelty evaluation is particularly brittle on the AI-generated ideas these systems are increasingly being asked to judge."},{"dimension":"summary","status":"pass","explanation":"The summary is understandable on its own. It names the actor and the study, explains that the judges assess whether research ideas are new, and gives the headline prompt-sensitivity result, the search and reasoning finding, and the purpose-built evaluator finding. It also notes the weak-label caveat. The question follows from the authors' call for evolving benchmarks. One small gap that does not block understanding: the 93.5% prompt told the model that exactly one idea per pair was novel.","excerpt":"One prompt told the model that reviewers had found exactly one idea in each pair to be novel and asked it to identify which one; the other simply asked which idea was novel."},{"dimension":"date","status":"pass","explanation":"The originalPublished value of 2026-10-27 matches the article's stated publication date of October 27, 2026.","excerpt":"Published\nOctober 27, 2026"},{"dimension":"support","status":"pass","explanation":"Each claim matches the source text: the release, the drop from 93.5% to 40.9% with changed verdicts on more than half of pairs, the finding that retrieval and reasoning did not consistently help, the purpose-built evaluators underperforming the baseline, and the weak labels. The limitations stay within the source and do not imply unsupported benefits or costs.","excerpt":"We assumed the generated ideas were less novel than their human-authored counterparts; these are weak labels, not verified judgments for every idea."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified link to the original Hugging Face article, which is the direct source of this reading.","excerpt":null}],"createdAt":"2026-11-04T15:02:29.812Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}