Zyskind, Guy · Nov 17, 2026
Language Models Make New Research Sound Old
Markdown · JSON · Original source
Guy Zyskind argues that (LLMs) can recognize that research questions remain unstudied yet fail to say so in ordinary answers, making new research sound partly studied. On 40 questions from newly accepted papers, GPT-6-Astra judged 98% unstudied but plainly reported that status in only 8% of ordinary answers. Zyskind says all six frontier systems had reporting gaps of 30–90 percentage points, while they identified studied questions correctly in 88–100% of ordinary answers when given the papers that answered them. The contrast matters for researchers using LLMs to assess whether a question is novel: a model’s judgment may be more accurate than its answer makes clear. The paper reports that combining a model’s judgment with separate checks of cited papers raised Astra’s clear reporting of open status to 95%, with improvements in all six systems. It also describes predicting reviewers’ novelty grades from models’ written ratings, without access to or internal activations. The evidence here is the paper’s abstract; it does not include the underlying evaluation details.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 17, 2026 (2026-11-17).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-19T08:32:49.579Z. First observed: 2026-11-19T05:01:39.299Z. Retrieved: 2026-11-19T08:31:56.519Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The supplied evidence is the abstract, not the paper’s full evaluation details; it does not provide enough information here to assess the study design or how the results extend beyond the tested questions.
Open question
How do the reported judgment and reporting gaps vary across questions beyond the 40 newly accepted papers tested?
Source-stated
Zyskind reports that LLMs often fail to clearly state that a research question is unstudied, despite judging it to be so.
Models recognize most of these overstatements when asked to check the papers' full texts separately.
Source-stated
For 40 questions, GPT-6-Astra judged 98% unstudied but plainly stated that status in only 8% of ordinary answers.
On 40 questions drawn from newly accepted papers, GPT-6-Astra correctly judges 98% to be unstudied, but states that status plainly in only 8% of ordinary answers.
Source-stated
The paper reports reporting gaps of 30–90 percentage points across six frontier systems and higher ordinary-answer accuracy for studied questions when answering papers were supplied.
All six frontier systems show reporting gaps of 30-90 percentage points.
Source-stated
Combining each model’s judgment with separate checks of cited papers raised Astra’s clear reporting of open status from 8% to 95%, with improvements across all six systems.
This raises Astra's clear reporting of open status from 8% to 95%, with improvements in all six systems.
Source-stated
The paper also reports predicting reviewers’ novelty grades from models’ written ratings using a rule learned from graded examples, without access to model weights or internal activations.
Unlike TPR, our approach needs no access to model weights or internal activations, so it also works with closed-weight frontier models.
Source and contributor context
Zyskind, Guy · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Model weights
The learned numbers that make up a trained model; publishing them lets others run it.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 9983001e-0576-4969-8fa4-c0c253902288.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-19T08:32:49.579Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 595d0ab77934eb9d01dfbfd9357b853c1dc39202add0cc6995f470ba43653a91. Source text hash: 832f7656c860c110e3502fbb5504e10ad4eeef0b70a72d80c4173da62f857a7d.
Revision and source-check history
Version 1 · 2026-11-19T08:32:49.579Z
Language Models Make New Research Sound Old
Guy Zyskind argues that (LLMs) can recognize that research questions remain unstudied yet fail to say so in ordinary answers, making new research sound partly studied. On 40 questions from newly accepted papers, GPT-6-Astra judged 98% unstudied but plainly reported that status in only 8% of ordinary answers. Zyskind says all six frontier systems had reporting gaps of 30–90 percentage points, while they identified studied questions correctly in 88–100% of ordinary answers when given the papers that answered them. The contrast matters for researchers using LLMs to assess whether a question is novel: a model’s judgment may be more accurate than its answer makes clear. The paper reports that combining a model’s judgment with separate checks of cited papers raised Astra’s clear reporting of open status to 95%, with improvements in all six systems. It also describes predicting reviewers’ novelty grades from models’ written ratings, without access to or internal activations. The evidence here is the paper’s abstract; it does not include the underlying evaluation details.
Original source
cs.AI · Article
Source details
Original source date: Nov 17, 2026 (2026-11-17).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-19T08:32:49.579Z. First observed: 2026-11-19T05:01:39.299Z. Retrieved: 2026-11-19T08:31:56.519Z.
Limits
The supplied evidence is the abstract, not the paper’s full evaluation details; it does not provide enough information here to assess the study design or how the results extend beyond the tested questions.
Open question
How do the reported judgment and reporting gaps vary across questions beyond the 40 newly accepted papers tested?
Source-stated
Zyskind reports that LLMs often fail to clearly state that a research question is unstudied, despite judging it to be so.
Models recognize most of these overstatements when asked to check the papers' full texts separately.
Source-stated
For 40 questions, GPT-6-Astra judged 98% unstudied but plainly stated that status in only 8% of ordinary answers.
On 40 questions drawn from newly accepted papers, GPT-6-Astra correctly judges 98% to be unstudied, but states that status plainly in only 8% of ordinary answers.
Source-stated
The paper reports reporting gaps of 30–90 percentage points across six frontier systems and higher ordinary-answer accuracy for studied questions when answering papers were supplied.
All six frontier systems show reporting gaps of 30-90 percentage points.
Source-stated
Combining each model’s judgment with separate checks of cited papers raised Astra’s clear reporting of open status from 8% to 95%, with improvements across all six systems.
This raises Astra's clear reporting of open status from 8% to 95%, with improvements in all six systems.
Source-stated
The paper also reports predicting reviewers’ novelty grades from models’ written ratings using a rule learned from graded examples, without access to model weights or internal activations.
Unlike TPR, our approach needs no access to model weights or internal activations, so it also works with closed-weight frontier models.
Source and contributor context
Zyskind, Guy · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Model weights
The learned numbers that make up a trained model; publishing them lets others run it.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 9983001e-0576-4969-8fa4-c0c253902288.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-19T08:32:49.579Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 595d0ab77934eb9d01dfbfd9357b853c1dc39202add0cc6995f470ba43653a91. Source text hash: 832f7656c860c110e3502fbb5504e10ad4eeef0b70a72d80c4173da62f857a7d.
uncertain · 2026-11-19T08:32:49.579Z · anthropic/claude-opus-5.5
Publication: 9983001e-0576-4969-8fa4-c0c253902288
attribution — pass: Guy Zyskind is the sole listed author, and the findings are consistently attributed to him or to the paper rather than presented as established fact.
Authors:Guy Zyskind
summary — uncertain: The reading is accurate but omits two parts of the abstract's argument. It drops that models 'overstate what related papers establish', which is the mechanism behind the title. It also drops the 22-model finding that higher capability goes with more accurate judgments and larger reporting gaps, a central, counterintuitive result. 'Frontier systems' and 'model weights or internal activations' also go unexplained. Smallest repair: add one sentence on overstating related papers and one on the 22-model capability/gap association, and add term tooltips.
Across 22 models given identical papers, higher capability is associated with more accurate judgments and larger reporting gaps.
date — pass: The originalPublished date, 2026-11-17, matches the arXiv v1 submission date. The feed date of 2026-11-19 is later and was correctly not used.
[Submitted on 17 Nov 2026]
support — pass: The figures (98% vs 8%, 30–90 points, 88–100%, 8% to 95%) and the novelty-grade method match the abstract. The inference that a model's judgment may be more accurate than its answer shows is hedged and follows from the reported gap. Minor issue: claim 1's excerpt concerns recognizing overstatements, not failing to state open status; the Astra 98%/8% excerpt would fit better. The source still supports the claim.
GPT-6-Astra correctly judges 98% to be unstudied, but states that status plainly in only 8% of ordinary answers
media — pass: The selected media is the verified direct link to the arXiv abstract page, which is the original source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A language model with many learned numerical settings, used to process and generate language.
Definition referenceThe learned numbers that make up a trained model; publishing them lets others run it.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referenceThe learned numbers that make up a trained model; publishing them lets others run it.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv