Zhang, Ke, Chundur, Sahchit, Owusu, Jerry Peprah, Qomi, Mohammad Javad Abdolhosseini, Raissi, Maziar · Oct 19, 2026
arXiv cs.AI: Where Retrieved Tool Output Costs LLM Agents Accuracy
Markdown · JSON · Original source
In their paper, Ke Zhang and coauthors argue that an agent can compute the right answer yet choose the wrong one when it retrieves a simulator’s report instead of reading the report in its . They tested agents using PHREEQC, a geochemistry simulator, and report sections accessed through an index. GPT-5.1 and GPT-5.2 lost accuracy under retrieval; Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost. The authors trace the GPT models’ errors not to changed simulations, but to evidence substitution: agents often skipped the report and answered from higher-precision output files, then misrounded to a neighboring option. Naming the authoritative output improved accuracy for the three GPT models, while asking for correct rounding did not improve results for GPT-5.1 and GPT-5.2. The authors’ practical point is that context-management choices should be evaluated by where accuracy is lost, not just by the final score. The reported evaluation uses PHREEQC and multiple-choice items; the evidence provided here does not establish how well the findings extend beyond that setting.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 19, 2026 (2026-10-19).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-18T18:48:23.339Z. First observed: 2026-10-20T04:19:11.022Z. Retrieved: 2026-11-18T18:47:32.182Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The evidence provided is the paper’s abstract and page, not the full paper; detailed methods and analyses are not available here. The reported evaluation uses PHREEQC and 158 execution-verified multiple-choice items. The evidence does not establish whether the findings generalize to other tools or tasks. The abstract reports model-specific results; it does not establish that the same effects occur across all agents or agent harnesses.
Open question
Would the same evidence substitution and accuracy losses appear with other scientific tools or tasks?
Source-stated
The paper reports that GPT-5.1 and GPT-5.2 lost accuracy when reading PHREEQC reports through a section index, while Claude Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost.
GPT-5.1 and GPT-5.2 lose 11.3 and 7.9 points of accuracy, while Claude Opus 4.6 and Claude Sonnet 4.6 show no detectable cost
Source-stated
The authors say that retrieval did not substantially change how often the two GPT models computed the keyed value; instead, they often answered from output files rather than the report and misrounded.
both GPT models compute the keyed value about as often as before; what changes is the evidence they answer from
Source-stated
The authors call the pattern “evidence substitution”: agents skip the report, rely on higher-precision output files, and select a neighboring answer after misrounding.
Under retrieval they often skip the report, answer from their own higher-precision output files, and misround to a neighboring option, a failure we call evidence substitution.
Source-stated
The authors report that naming the authoritative output improved accuracy for the three GPT models, while asking for correct rounding did not improve results for GPT-5.1 and GPT-5.2.
Naming the authoritative output raises accuracy by 6.3-7.9 points over plain retrieval for the three GPT models
Source-stated
The authors recommend evaluating context-management choices by identifying where accuracy is lost, rather than judging only the final accuracy.
Context-management choices in agent harnesses should be judged by where accuracy is lost, not only by final accuracy.
Source and contributor context
Zhang, Ke, Chundur, Sahchit, Owusu, Jerry Peprah, Qomi, Mohammad Javad Abdolhosseini, Raissi, Maziar · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Context window
How much text a model can take in and consider at one time.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: b0079431-068e-4ea2-8ae0-47852db683b2.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T18:48:23.339Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 38c470a91256bdcf2672a522b394738be0ced851e7f9362c38bd2705ee398a6b. Source text hash: 65a26635581c3b61b8983ea6ff388846c03fc2483e3936d9936d4a6695636b62.
Revision and source-check history
Version 1 · 2026-11-18T18:48:23.339Z
arXiv cs.AI: Where Retrieved Tool Output Costs LLM Agents Accuracy
In their paper, Ke Zhang and coauthors argue that an agent can compute the right answer yet choose the wrong one when it retrieves a simulator’s report instead of reading the report in its . They tested agents using PHREEQC, a geochemistry simulator, and report sections accessed through an index. GPT-5.1 and GPT-5.2 lost accuracy under retrieval; Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost. The authors trace the GPT models’ errors not to changed simulations, but to evidence substitution: agents often skipped the report and answered from higher-precision output files, then misrounded to a neighboring option. Naming the authoritative output improved accuracy for the three GPT models, while asking for correct rounding did not improve results for GPT-5.1 and GPT-5.2. The authors’ practical point is that context-management choices should be evaluated by where accuracy is lost, not just by the final score. The reported evaluation uses PHREEQC and multiple-choice items; the evidence provided here does not establish how well the findings extend beyond that setting.
Original source
cs.AI · Article
Source details
Original source date: Oct 19, 2026 (2026-10-19).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-18T18:48:23.339Z. First observed: 2026-10-20T04:19:11.022Z. Retrieved: 2026-11-18T18:47:32.182Z.
Limits
The evidence provided is the paper’s abstract and page, not the full paper; detailed methods and analyses are not available here. The reported evaluation uses PHREEQC and 158 execution-verified multiple-choice items. The evidence does not establish whether the findings generalize to other tools or tasks. The abstract reports model-specific results; it does not establish that the same effects occur across all agents or agent harnesses.
Open question
Would the same evidence substitution and accuracy losses appear with other scientific tools or tasks?
Source-stated
The paper reports that GPT-5.1 and GPT-5.2 lost accuracy when reading PHREEQC reports through a section index, while Claude Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost.
GPT-5.1 and GPT-5.2 lose 11.3 and 7.9 points of accuracy, while Claude Opus 4.6 and Claude Sonnet 4.6 show no detectable cost
Source-stated
The authors say that retrieval did not substantially change how often the two GPT models computed the keyed value; instead, they often answered from output files rather than the report and misrounded.
both GPT models compute the keyed value about as often as before; what changes is the evidence they answer from
Source-stated
The authors call the pattern “evidence substitution”: agents skip the report, rely on higher-precision output files, and select a neighboring answer after misrounding.
Under retrieval they often skip the report, answer from their own higher-precision output files, and misround to a neighboring option, a failure we call evidence substitution.
Source-stated
The authors report that naming the authoritative output improved accuracy for the three GPT models, while asking for correct rounding did not improve results for GPT-5.1 and GPT-5.2.
Naming the authoritative output raises accuracy by 6.3-7.9 points over plain retrieval for the three GPT models
Source-stated
The authors recommend evaluating context-management choices by identifying where accuracy is lost, rather than judging only the final accuracy.
Context-management choices in agent harnesses should be judged by where accuracy is lost, not only by final accuracy.
Source and contributor context
Zhang, Ke, Chundur, Sahchit, Owusu, Jerry Peprah, Qomi, Mohammad Javad Abdolhosseini, Raissi, Maziar · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Context window
How much text a model can take in and consider at one time.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: b0079431-068e-4ea2-8ae0-47852db683b2.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T18:48:23.339Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 38c470a91256bdcf2672a522b394738be0ced851e7f9362c38bd2705ee398a6b. Source text hash: 65a26635581c3b61b8983ea6ff388846c03fc2483e3936d9936d4a6695636b62.
uncertain · 2026-11-18T18:48:23.339Z · anthropic/claude-opus-5.5
Publication: b0079431-068e-4ea2-8ae0-47852db683b2
attribution — pass: The reading attributes the work to Ke Zhang and coauthors on arXiv cs.AI. It keeps the findings and recommendation framed as the authors' reports and argument rather than established fact.
Authors:Ke Zhang, Sahchit Chundur, Jerry Peprah Owusu, Mohammad Javad Abdolhosseini Qomi, Maziar Raissi
summary — uncertain: The summary and claims refer to 'the three GPT models', but only GPT-5.1 and GPT-5.2 are introduced. GPT-5.4 is missing, so readers cannot tell which third model is meant. The reading also omits the GPT-5.4 nuance: more frequent correct simulations masked an evidence-use loss of about ten points, leaving a net change indistinguishable from zero. Smallest repair: add one sentence on the GPT-5.4 result. 'Context window' also appears without a plain-language gloss.
For GPT-5.4, more frequent correct simulations mask an evidence-use loss of about ten points, leaving a net change indistinguishable from zero.
date — pass: originalPublished is 2026-10-19, which matches the v1 submission date. It does not use the 16 Nov 2026 revision or the feed date.
Submitted on 19 Oct 2026 (v1), last revised 16 Nov 2026 (this version, v2)
support — uncertain: The abstract says that, for GPT-5.1 and GPT-5.2, inlining outputs 'works as well' (as naming the authoritative output), 'while asking for correct rounding does not.' This most naturally means rounding requests did not work as well as naming. The reading's claim that they 'did not improve results' may overstate this. Smallest repair: say asking for correct rounding did not work as well. The other claims are supported, and the limitations are appropriately framed.
for GPT-5.1 and GPT-5.2, inlining outputs that fit works as well, while asking for correct rounding does not.
media — pass: The selected media is the verified direct link to the arXiv abstract page, which is the original source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A language model with many learned numerical settings, used to process and generate language.
Definition referenceHow much text a model can take in and consider at one time.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceHow much text a model can take in and consider at one time.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition reference