# arXiv cs.AI: Where Retrieved Tool Output Costs LLM Agents Accuracy

[Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-where-retrieved-tool-output-costs-llm-agents-c69e72e9)

In their paper, Ke Zhang and coauthors argue that an LLM agent can compute the right answer yet choose the wrong one when it retrieves a simulator’s report instead of reading the report in its context window. They tested agents using PHREEQC, a geochemistry simulator, and report sections accessed through an index. GPT-5.1 and GPT-5.2 lost accuracy under retrieval; Claude Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost. The authors trace the GPT models’ errors not to changed simulations, but to evidence substitution: agents often skipped the report and answered from higher-precision output files, then misrounded to a neighboring option. Naming the authoritative output improved accuracy for the three GPT models, while asking for correct rounding did not improve results for GPT-5.1 and GPT-5.2. The authors’ practical point is that context-management choices should be evaluated by where accuracy is lost, not just by the final score. The reported evaluation uses PHREEQC and multiple-choice items; the evidence provided here does not establish how well the findings extend beyond that setting.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv cs.AI · Article

Original source date: Oct 19, 2026 (2026-10-19).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-18T18:48:23.339Z. First observed: 2026-10-20T04:19:11.022Z. Retrieved: 2026-11-18T18:47:32.182Z.

- [Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy](https://arxiv.org/abs/2610.26836)

## Limits

The evidence provided is the paper’s abstract and arXiv page, not the full paper; detailed methods and analyses are not available here. The reported evaluation uses PHREEQC and 158 execution-verified multiple-choice items. The evidence does not establish whether the findings generalize to other tools or tasks. The abstract reports model-specific results; it does not establish that the same effects occur across all LLM agents or agent harnesses.


## Open question

Would the same evidence substitution and accuracy losses appear with other scientific tools or tasks?


## Source-stated

The paper reports that GPT-5.1 and GPT-5.2 lost accuracy when reading PHREEQC reports through a section index, while Claude Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost.

> GPT-5.1 and GPT-5.2 lose 11.3 and 7.9 points of accuracy, while Claude Opus 4.6 and Claude Sonnet 4.6 show no detectable cost

- [Supporting source](https://arxiv.org/abs/2610.26836)

## Source-stated

The authors say that retrieval did not substantially change how often the two GPT models computed the keyed value; instead, they often answered from output files rather than the report and misrounded.

> both GPT models compute the keyed value about as often as before; what changes is the evidence they answer from

- [Supporting source](https://arxiv.org/abs/2610.26836)

## Source-stated

The authors call the pattern “evidence substitution”: agents skip the report, rely on higher-precision output files, and select a neighboring answer after misrounding.

> Under retrieval they often skip the report, answer from their own higher-precision output files, and misround to a neighboring option, a failure we call evidence substitution.

- [Supporting source](https://arxiv.org/abs/2610.26836)

## Source-stated

The authors report that naming the authoritative output improved accuracy for the three GPT models, while asking for correct rounding did not improve results for GPT-5.1 and GPT-5.2.

> Naming the authoritative output raises accuracy by 6.3-7.9 points over plain retrieval for the three GPT models

- [Supporting source](https://arxiv.org/abs/2610.26836)

## Source-stated

The authors recommend evaluating context-management choices by identifying where accuracy is lost, rather than judging only the final accuracy.

> Context-management choices in agent harnesses should be judged by where accuracy is lost, not only by final accuracy.

- [Supporting source](https://arxiv.org/abs/2610.26836)

## Source and contributor context

Zhang, Ke, Chundur, Sahchit, Owusu, Jerry Peprah, Qomi, Mohammad Javad Abdolhosseini, Raissi, Maziar · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## Context window

How much text a model can take in and consider at one time.

- [Definition reference](https://developers.google.com/machine-learning/glossary#context-window)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Claude

Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.

- [Explore this context](/entities/claude)
- [Reference](https://claude.com/product/overview)
- [Reference](https://www.anthropic.com/news)
- [Reference](https://www.anthropic.com/research/team/economics)

## Editorial provenance

Version 1. Publication ID: b0079431-068e-4ea2-8ae0-47852db683b2.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T18:48:23.339Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 38c470a91256bdcf2672a522b394738be0ced851e7f9362c38bd2705ee398a6b. Source text hash: 65a26635581c3b61b8983ea6ff388846c03fc2483e3936d9936d4a6695636b62.


## Revision and source-check history

### Version 1 — 2026-11-18T18:48:23.339Z

In their paper, Ke Zhang and coauthors argue that an LLM agent can compute the right answer yet choose the wrong one when it retrieves a simulator’s report instead of reading the report in its context window. They tested agents using PHREEQC, a geochemistry simulator, and report sections accessed through an index. GPT-5.1 and GPT-5.2 lost accuracy under retrieval; Claude Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost. The authors trace the GPT models’ errors not to changed simulations, but to evidence substitution: agents often skipped the report and answered from higher-precision output files, then misrounded to a neighboring option. Naming the authoritative output improved accuracy for the three GPT models, while asking for correct rounding did not improve results for GPT-5.1 and GPT-5.2. The authors’ practical point is that context-management choices should be evaluated by where accuracy is lost, not just by the final score. The reported evaluation uses PHREEQC and multiple-choice items; the evidence provided here does not establish how well the findings extend beyond that setting.

## Original source

arXiv cs.AI · Article

Original source date: Oct 19, 2026 (2026-10-19).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-18T18:48:23.339Z. First observed: 2026-10-20T04:19:11.022Z. Retrieved: 2026-11-18T18:47:32.182Z.

- [Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy](https://arxiv.org/abs/2610.26836)

## Limits

The evidence provided is the paper’s abstract and arXiv page, not the full paper; detailed methods and analyses are not available here. The reported evaluation uses PHREEQC and 158 execution-verified multiple-choice items. The evidence does not establish whether the findings generalize to other tools or tasks. The abstract reports model-specific results; it does not establish that the same effects occur across all LLM agents or agent harnesses.


## Open question

Would the same evidence substitution and accuracy losses appear with other scientific tools or tasks?


## Source-stated

The paper reports that GPT-5.1 and GPT-5.2 lost accuracy when reading PHREEQC reports through a section index, while Claude Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost.

> GPT-5.1 and GPT-5.2 lose 11.3 and 7.9 points of accuracy, while Claude Opus 4.6 and Claude Sonnet 4.6 show no detectable cost

- [Supporting source](https://arxiv.org/abs/2610.26836)

## Source-stated

The authors say that retrieval did not substantially change how often the two GPT models computed the keyed value; instead, they often answered from output files rather than the report and misrounded.

> both GPT models compute the keyed value about as often as before; what changes is the evidence they answer from

- [Supporting source](https://arxiv.org/abs/2610.26836)

## Source-stated

The authors call the pattern “evidence substitution”: agents skip the report, rely on higher-precision output files, and select a neighboring answer after misrounding.

> Under retrieval they often skip the report, answer from their own higher-precision output files, and misround to a neighboring option, a failure we call evidence substitution.

- [Supporting source](https://arxiv.org/abs/2610.26836)

## Source-stated

The authors report that naming the authoritative output improved accuracy for the three GPT models, while asking for correct rounding did not improve results for GPT-5.1 and GPT-5.2.

> Naming the authoritative output raises accuracy by 6.3-7.9 points over plain retrieval for the three GPT models

- [Supporting source](https://arxiv.org/abs/2610.26836)

## Source-stated

The authors recommend evaluating context-management choices by identifying where accuracy is lost, rather than judging only the final accuracy.

> Context-management choices in agent harnesses should be judged by where accuracy is lost, not only by final accuracy.

- [Supporting source](https://arxiv.org/abs/2610.26836)

## Source and contributor context

Zhang, Ke, Chundur, Sahchit, Owusu, Jerry Peprah, Qomi, Mohammad Javad Abdolhosseini, Raissi, Maziar · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## Context window

How much text a model can take in and consider at one time.

- [Definition reference](https://developers.google.com/machine-learning/glossary#context-window)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Claude

Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.

- [Explore this context](/entities/claude)
- [Reference](https://claude.com/product/overview)
- [Reference](https://www.anthropic.com/news)
- [Reference](https://www.anthropic.com/research/team/economics)

## Editorial provenance

Version 1. Publication ID: b0079431-068e-4ea2-8ae0-47852db683b2.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T18:48:23.339Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 38c470a91256bdcf2672a522b394738be0ced851e7f9362c38bd2705ee398a6b. Source text hash: 65a26635581c3b61b8983ea6ff388846c03fc2483e3936d9936d4a6695636b62.


### uncertain — 2026-11-18T18:48:23.339Z — anthropic/claude-opus-5.5

Publication: b0079431-068e-4ea2-8ae0-47852db683b2

attribution — pass: The reading attributes the work to Ke Zhang and coauthors on arXiv cs.AI. It keeps the findings and recommendation framed as the authors' reports and argument rather than established fact.

> Authors:Ke Zhang, Sahchit Chundur, Jerry Peprah Owusu, Mohammad Javad Abdolhosseini Qomi, Maziar Raissi

summary — uncertain: The summary and claims refer to 'the three GPT models', but only GPT-5.1 and GPT-5.2 are introduced. GPT-5.4 is missing, so readers cannot tell which third model is meant. The reading also omits the GPT-5.4 nuance: more frequent correct simulations masked an evidence-use loss of about ten points, leaving a net change indistinguishable from zero. Smallest repair: add one sentence on the GPT-5.4 result. 'Context window' also appears without a plain-language gloss.

> For GPT-5.4, more frequent correct simulations mask an evidence-use loss of about ten points, leaving a net change indistinguishable from zero.

date — pass: originalPublished is 2026-10-19, which matches the v1 submission date. It does not use the 16 Nov 2026 revision or the feed date.

> Submitted on 19 Oct 2026 (v1), last revised 16 Nov 2026 (this version, v2)

support — uncertain: The abstract says that, for GPT-5.1 and GPT-5.2, inlining outputs 'works as well' (as naming the authoritative output), 'while asking for correct rounding does not.' This most naturally means rounding requests did not work as well as naming. The reading's claim that they 'did not improve results' may overstate this. Smallest repair: say asking for correct rounding did not work as well. The other claims are supported, and the limitations are appropriately framed.

> for GPT-5.1 and GPT-5.2, inlining outputs that fit works as well, while asking for correct rounding does not.

media — pass: The selected media is the verified direct link to the arXiv abstract page, which is the original source.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/c69e72e9e308f1f46f5dafa9cfdf243439fde68e8f6557d6faf508b4693e754f/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-where-retrieved-tool-output-costs-llm-agents-c69e72e9.json)
