{"id":"c69e72e9e308f1f46f5dafa9cfdf243439fde68e8f6557d6faf508b4693e754f","title":"arXiv cs.AI: Where Retrieved Tool Output Costs LLM Agents Accuracy","argument":"In their paper, Ke Zhang and coauthors argue that an LLM agent can compute the right answer yet choose the wrong one when it retrieves a simulator’s report instead of reading the report in its context window. They tested agents using PHREEQC, a geochemistry simulator, and report sections accessed through an index. GPT-5.1 and GPT-5.2 lost accuracy under retrieval; Claude Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost. The authors trace the GPT models’ errors not to changed simulations, but to evidence substitution: agents often skipped the report and answered from higher-precision output files, then misrounded to a neighboring option. Naming the authoritative output improved accuracy for the three GPT models, while asking for correct rounding did not improve results for GPT-5.1 and GPT-5.2. The authors’ practical point is that context-management choices should be evaluated by where accuracy is lost, not just by the final score. The reported evaluation uses PHREEQC and multiple-choice items; the evidence provided here does not establish how well the findings extend beyond that setting.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence provided is the paper’s abstract and arXiv page, not the full paper; detailed methods and analyses are not available here. The reported evaluation uses PHREEQC and 158 execution-verified multiple-choice items. The evidence does not establish whether the findings generalize to other tools or tasks. The abstract reports model-specific results; it does not establish that the same effects occur across all LLM agents or agent harnesses.","speaker":{"name":"Zhang, Ke, Chundur, Sahchit, Owusu, Jerry Peprah, Qomi, Mohammad Javad Abdolhosseini, Raissi, Maziar","roleAtSource":"Article author"},"source":{"title":"Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.26836","publishedAt":"2026-10-19","dateBasis":"Date basis not provided."},"evidence":[{"text":"The paper reports that GPT-5.1 and GPT-5.2 lost accuracy when reading PHREEQC reports through a section index, while Claude Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost.","excerpt":"GPT-5.1 and GPT-5.2 lose 11.3 and 7.9 points of accuracy, while Claude Opus 4.6 and Claude Sonnet 4.6 show no detectable cost","url":"https://arxiv.org/abs/2610.26836","relationship":"Source-stated"},{"text":"The authors say that retrieval did not substantially change how often the two GPT models computed the keyed value; instead, they often answered from output files rather than the report and misrounded.","excerpt":"both GPT models compute the keyed value about as often as before; what changes is the evidence they answer from","url":"https://arxiv.org/abs/2610.26836","relationship":"Source-stated"},{"text":"The authors call the pattern “evidence substitution”: agents skip the report, rely on higher-precision output files, and select a neighboring answer after misrounding.","excerpt":"Under retrieval they often skip the report, answer from their own higher-precision output files, and misround to a neighboring option, a failure we call evidence substitution.","url":"https://arxiv.org/abs/2610.26836","relationship":"Source-stated"},{"text":"The authors report that naming the authoritative output improved accuracy for the three GPT models, while asking for correct rounding did not improve results for GPT-5.1 and GPT-5.2.","excerpt":"Naming the authoritative output raises accuracy by 6.3-7.9 points over plain retrieval for the three GPT models","url":"https://arxiv.org/abs/2610.26836","relationship":"Source-stated"},{"text":"The authors recommend evaluating context-management choices by identifying where accuracy is lost, rather than judging only the final accuracy.","excerpt":"Context-management choices in agent harnesses should be judged by where accuracy is lost, not only by final accuracy.","url":"https://arxiv.org/abs/2610.26836","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.26836","checkedAt":"2026-11-18T19:52:10.547Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Would the same evidence substitution and accuracy losses appear with other scientific tools or tasks?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Where Retrieved Tool Output Costs LLM Agents Accuracy","summary":"In their paper, Ke Zhang and coauthors argue that an LLM agent can compute the right answer yet choose the wrong one when it retrieves a simulator’s report instead of reading the report in its context window. They tested agents using PHREEQC, a geochemistry simulator, and report sections accessed through an index. GPT-5.1 and GPT-5.2 lost accuracy under retrieval; Claude Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost. The authors trace the GPT models’ errors not to changed simulations, but to evidence substitution: agents often skipped the report and answered from higher-precision output files, then misrounded to a neighboring option. Naming the authoritative output improved accuracy for the three GPT models, while asking for correct rounding did not improve results for GPT-5.1 and GPT-5.2. The authors’ practical point is that context-management choices should be evaluated by where accuracy is lost, not just by the final score. The reported evaluation uses PHREEQC and multiple-choice items; the evidence provided here does not establish how well the findings extend beyond that setting.","question":"Would the same evidence substitution and accuracy losses appear with other scientific tools or tasks?","claims":[{"text":"The paper reports that GPT-5.1 and GPT-5.2 lost accuracy when reading PHREEQC reports through a section index, while Claude Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost.","excerpt":"GPT-5.1 and GPT-5.2 lose 11.3 and 7.9 points of accuracy, while Claude Opus 4.6 and Claude Sonnet 4.6 show no detectable cost","support":"source-stated"},{"text":"The authors say that retrieval did not substantially change how often the two GPT models computed the keyed value; instead, they often answered from output files rather than the report and misrounded.","excerpt":"both GPT models compute the keyed value about as often as before; what changes is the evidence they answer from","support":"source-stated"},{"text":"The authors call the pattern “evidence substitution”: agents skip the report, rely on higher-precision output files, and select a neighboring answer after misrounding.","excerpt":"Under retrieval they often skip the report, answer from their own higher-precision output files, and misround to a neighboring option, a failure we call evidence substitution.","support":"source-stated"},{"text":"The authors report that naming the authoritative output improved accuracy for the three GPT models, while asking for correct rounding did not improve results for GPT-5.1 and GPT-5.2.","excerpt":"Naming the authoritative output raises accuracy by 6.3-7.9 points over plain retrieval for the three GPT models","support":"source-stated"},{"text":"The authors recommend evaluating context-management choices by identifying where accuracy is lost, rather than judging only the final accuracy.","excerpt":"Context-management choices in agent harnesses should be judged by where accuracy is lost, not only by final accuracy.","support":"source-stated"}],"limitations":["The evidence provided is the paper’s abstract and arXiv page, not the full paper; detailed methods and analyses are not available here.","The reported evaluation uses PHREEQC and 158 execution-verified multiple-choice items. The evidence does not establish whether the findings generalize to other tools or tasks.","The abstract reports model-specific results; it does not establish that the same effects occur across all LLM agents or agent harnesses."],"mediaId":"029806d18c63cad7567a0597f6504f57c0dfe5075c2209fcd35e85e0e0d6a818","correctionReason":null,"id":"c69e72e9e308f1f46f5dafa9cfdf243439fde68e8f6557d6faf508b4693e754f","publicationId":"b0079431-068e-4ea2-8ae0-47852db683b2","version":1,"sourceUrl":"https://arxiv.org/abs/2610.26836","sourceTitle":"Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-19","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-20T04:19:11.022Z","createdAt":"2026-11-18T18:48:23.339Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"029806d18c63cad7567a0597f6504f57c0dfe5075c2209fcd35e85e0e0d6a818","kind":"article","url":"https://arxiv.org/abs/2610.26836","sourceUrl":"https://arxiv.org/abs/2610.26836","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-18T18:47:32.263Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.26836","checkedAt":"2026-11-18T19:52:10.547Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"029806d18c63cad7567a0597f6504f57c0dfe5075c2209fcd35e85e0e0d6a818","confidence":0.8400000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"cd32b8448ff55a63b3ad6c7902ed7faf68fdfc5a4843a59ad5b5a756912a0edb","decidedAt":"2026-11-18T18:47:32.888Z"},"provenance":{"textHash":"65a26635581c3b61b8983ea6ff388846c03fc2483e3936d9936d4a6695636b62","observationId":"38c470a91256bdcf2672a522b394738be0ced851e7f9362c38bd2705ee398a6b","retrievedAt":"2026-11-18T18:47:32.182Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"23455e21-ccd4-4464-8566-c069896ae7de"},"reviewer":{"model":"gpt-6-luna","runId":"e4ab5023-d009-4fb5-8d87-3ab4b39d8434"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-18T18:48:23.339Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Zhang, Ke","url":null,"kind":"unknown"},{"name":"Chundur, Sahchit","url":null,"kind":"unknown"},{"name":"Owusu, Jerry Peprah","url":null,"kind":"unknown"},{"name":"Qomi, Mohammad Javad Abdolhosseini","url":null,"kind":"unknown"},{"name":"Raissi, Maziar","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Zhang, Ke","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Ke","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.26836","locator":"head/meta[name=author|citation_author]"},"assertionId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:52:10.547Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc","sourceUrl":"https://arxiv.org/abs/2610.26836","contentSha256":"630bd24732d2a21fd6126c1f57719e04500c58ec2627388eff1273859bc993c8","textHash":"65a26635581c3b61b8983ea6ff388846c03fc2483e3936d9936d4a6695636b62","observedAt":"2026-11-18T19:52:10.547Z"}},{"actor":{"name":"Chundur, Sahchit","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chundur, Sahchit","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.26836","locator":"head/meta[name=author|citation_author]"},"assertionId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:52:10.547Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc","sourceUrl":"https://arxiv.org/abs/2610.26836","contentSha256":"630bd24732d2a21fd6126c1f57719e04500c58ec2627388eff1273859bc993c8","textHash":"65a26635581c3b61b8983ea6ff388846c03fc2483e3936d9936d4a6695636b62","observedAt":"2026-11-18T19:52:10.547Z"}},{"actor":{"name":"Owusu, Jerry Peprah","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Owusu, Jerry Peprah","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.26836","locator":"head/meta[name=author|citation_author]"},"assertionId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:52:10.547Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc","sourceUrl":"https://arxiv.org/abs/2610.26836","contentSha256":"630bd24732d2a21fd6126c1f57719e04500c58ec2627388eff1273859bc993c8","textHash":"65a26635581c3b61b8983ea6ff388846c03fc2483e3936d9936d4a6695636b62","observedAt":"2026-11-18T19:52:10.547Z"}},{"actor":{"name":"Qomi, Mohammad Javad Abdolhosseini","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Qomi, Mohammad Javad Abdolhosseini","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.26836","locator":"head/meta[name=author|citation_author]"},"assertionId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:52:10.547Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc","sourceUrl":"https://arxiv.org/abs/2610.26836","contentSha256":"630bd24732d2a21fd6126c1f57719e04500c58ec2627388eff1273859bc993c8","textHash":"65a26635581c3b61b8983ea6ff388846c03fc2483e3936d9936d4a6695636b62","observedAt":"2026-11-18T19:52:10.547Z"}},{"actor":{"name":"Raissi, Maziar","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Raissi, Maziar","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.26836","locator":"head/meta[name=author|citation_author]"},"assertionId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:52:10.547Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc","sourceUrl":"https://arxiv.org/abs/2610.26836","contentSha256":"630bd24732d2a21fd6126c1f57719e04500c58ec2627388eff1273859bc993c8","textHash":"65a26635581c3b61b8983ea6ff388846c03fc2483e3936d9936d4a6695636b62","observedAt":"2026-11-18T19:52:10.547Z"}}],"participationCoverage":{"total":5,"returned":5,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.26836","snapshotId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc","contentSha256":"630bd24732d2a21fd6126c1f57719e04500c58ec2627388eff1273859bc993c8","textHash":"65a26635581c3b61b8983ea6ff388846c03fc2483e3936d9936d4a6695636b62","observedAt":"2026-11-18T19:52:10.547Z","basis":"publisher-page-metadata","sourceLinks":45,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/c69e72e9e308f1f46f5dafa9cfdf243439fde68e8f6557d6faf508b4693e754f/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-where-retrieved-tool-output-costs-llm-agents-c69e72e9","formats":{"html":"/readings/arxiv-cs-ai-where-retrieved-tool-output-costs-llm-agents-c69e72e9","markdown":"/readings/arxiv-cs-ai-where-retrieved-tool-output-costs-llm-agents-c69e72e9.md","json":"/readings/arxiv-cs-ai-where-retrieved-tool-output-costs-llm-agents-c69e72e9.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"},{"id":"term-context-window","name":"Context window","explanation":"How much text a model can take in and consider at one time.","referenceUrl":"https://developers.google.com/machine-learning/glossary#context-window"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"},{"id":"claude","name":"Claude","kind":"model","context":"Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.","referenceUrls":["https://claude.com/product/overview","https://www.anthropic.com/news","https://www.anthropic.com/research/team/economics"],"path":"/entities/claude","json":"/entities/claude.json","markdown":"/entities/claude.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"c69e72e9e308f1f46f5dafa9cfdf243439fde68e8f6557d6faf508b4693e754f","revision":"b0079431-068e-4ea2-8ae0-47852db683b2","revisionKind":"publication-id","title":"arXiv cs.AI: Where Retrieved Tool Output Costs LLM Agents Accuracy","url":"https://escapemeridian.com/readings/arxiv-cs-ai-where-retrieved-tool-output-costs-llm-agents-c69e72e9"},"source":{"type":"source","id":"https://arxiv.org/abs/2610.26836","revision":"65a26635581c3b61b8983ea6ff388846c03fc2483e3936d9936d4a6695636b62","revisionKind":"sha256-source-text","title":"Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy","url":"https://arxiv.org/abs/2610.26836"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"c69e72e9e308f1f46f5dafa9cfdf243439fde68e8f6557d6faf508b4693e754f","title":"arXiv cs.AI: Where Retrieved Tool Output Costs LLM Agents Accuracy","argument":"In their paper, Ke Zhang and coauthors argue that an LLM agent can compute the right answer yet choose the wrong one when it retrieves a simulator’s report instead of reading the report in its context window. They tested agents using PHREEQC, a geochemistry simulator, and report sections accessed through an index. GPT-5.1 and GPT-5.2 lost accuracy under retrieval; Claude Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost. The authors trace the GPT models’ errors not to changed simulations, but to evidence substitution: agents often skipped the report and answered from higher-precision output files, then misrounded to a neighboring option. Naming the authoritative output improved accuracy for the three GPT models, while asking for correct rounding did not improve results for GPT-5.1 and GPT-5.2. The authors’ practical point is that context-management choices should be evaluated by where accuracy is lost, not just by the final score. The reported evaluation uses PHREEQC and multiple-choice items; the evidence provided here does not establish how well the findings extend beyond that setting.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence provided is the paper’s abstract and arXiv page, not the full paper; detailed methods and analyses are not available here. The reported evaluation uses PHREEQC and 158 execution-verified multiple-choice items. The evidence does not establish whether the findings generalize to other tools or tasks. The abstract reports model-specific results; it does not establish that the same effects occur across all LLM agents or agent harnesses.","speaker":{"name":"Zhang, Ke, Chundur, Sahchit, Owusu, Jerry Peprah, Qomi, Mohammad Javad Abdolhosseini, Raissi, Maziar","roleAtSource":"Article author"},"source":{"title":"Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.26836","publishedAt":"2026-10-19","dateBasis":"Date basis not provided."},"evidence":[{"text":"The paper reports that GPT-5.1 and GPT-5.2 lost accuracy when reading PHREEQC reports through a section index, while Claude Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost.","excerpt":"GPT-5.1 and GPT-5.2 lose 11.3 and 7.9 points of accuracy, while Claude Opus 4.6 and Claude Sonnet 4.6 show no detectable cost","url":"https://arxiv.org/abs/2610.26836","relationship":"Source-stated"},{"text":"The authors say that retrieval did not substantially change how often the two GPT models computed the keyed value; instead, they often answered from output files rather than the report and misrounded.","excerpt":"both GPT models compute the keyed value about as often as before; what changes is the evidence they answer from","url":"https://arxiv.org/abs/2610.26836","relationship":"Source-stated"},{"text":"The authors call the pattern “evidence substitution”: agents skip the report, rely on higher-precision output files, and select a neighboring answer after misrounding.","excerpt":"Under retrieval they often skip the report, answer from their own higher-precision output files, and misround to a neighboring option, a failure we call evidence substitution.","url":"https://arxiv.org/abs/2610.26836","relationship":"Source-stated"},{"text":"The authors report that naming the authoritative output improved accuracy for the three GPT models, while asking for correct rounding did not improve results for GPT-5.1 and GPT-5.2.","excerpt":"Naming the authoritative output raises accuracy by 6.3-7.9 points over plain retrieval for the three GPT models","url":"https://arxiv.org/abs/2610.26836","relationship":"Source-stated"},{"text":"The authors recommend evaluating context-management choices by identifying where accuracy is lost, rather than judging only the final accuracy.","excerpt":"Context-management choices in agent harnesses should be judged by where accuracy is lost, not only by final accuracy.","url":"https://arxiv.org/abs/2610.26836","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.26836","checkedAt":"2026-11-18T19:52:10.547Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Would the same evidence substitution and accuracy losses appear with other scientific tools or tasks?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Where Retrieved Tool Output Costs LLM Agents Accuracy","summary":"In their paper, Ke Zhang and coauthors argue that an LLM agent can compute the right answer yet choose the wrong one when it retrieves a simulator’s report instead of reading the report in its context window. They tested agents using PHREEQC, a geochemistry simulator, and report sections accessed through an index. GPT-5.1 and GPT-5.2 lost accuracy under retrieval; Claude Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost. The authors trace the GPT models’ errors not to changed simulations, but to evidence substitution: agents often skipped the report and answered from higher-precision output files, then misrounded to a neighboring option. Naming the authoritative output improved accuracy for the three GPT models, while asking for correct rounding did not improve results for GPT-5.1 and GPT-5.2. The authors’ practical point is that context-management choices should be evaluated by where accuracy is lost, not just by the final score. The reported evaluation uses PHREEQC and multiple-choice items; the evidence provided here does not establish how well the findings extend beyond that setting.","question":"Would the same evidence substitution and accuracy losses appear with other scientific tools or tasks?","claims":[{"text":"The paper reports that GPT-5.1 and GPT-5.2 lost accuracy when reading PHREEQC reports through a section index, while Claude Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost.","excerpt":"GPT-5.1 and GPT-5.2 lose 11.3 and 7.9 points of accuracy, while Claude Opus 4.6 and Claude Sonnet 4.6 show no detectable cost","support":"source-stated"},{"text":"The authors say that retrieval did not substantially change how often the two GPT models computed the keyed value; instead, they often answered from output files rather than the report and misrounded.","excerpt":"both GPT models compute the keyed value about as often as before; what changes is the evidence they answer from","support":"source-stated"},{"text":"The authors call the pattern “evidence substitution”: agents skip the report, rely on higher-precision output files, and select a neighboring answer after misrounding.","excerpt":"Under retrieval they often skip the report, answer from their own higher-precision output files, and misround to a neighboring option, a failure we call evidence substitution.","support":"source-stated"},{"text":"The authors report that naming the authoritative output improved accuracy for the three GPT models, while asking for correct rounding did not improve results for GPT-5.1 and GPT-5.2.","excerpt":"Naming the authoritative output raises accuracy by 6.3-7.9 points over plain retrieval for the three GPT models","support":"source-stated"},{"text":"The authors recommend evaluating context-management choices by identifying where accuracy is lost, rather than judging only the final accuracy.","excerpt":"Context-management choices in agent harnesses should be judged by where accuracy is lost, not only by final accuracy.","support":"source-stated"}],"limitations":["The evidence provided is the paper’s abstract and arXiv page, not the full paper; detailed methods and analyses are not available here.","The reported evaluation uses PHREEQC and 158 execution-verified multiple-choice items. The evidence does not establish whether the findings generalize to other tools or tasks.","The abstract reports model-specific results; it does not establish that the same effects occur across all LLM agents or agent harnesses."],"mediaId":"029806d18c63cad7567a0597f6504f57c0dfe5075c2209fcd35e85e0e0d6a818","correctionReason":null,"id":"c69e72e9e308f1f46f5dafa9cfdf243439fde68e8f6557d6faf508b4693e754f","publicationId":"b0079431-068e-4ea2-8ae0-47852db683b2","version":1,"sourceUrl":"https://arxiv.org/abs/2610.26836","sourceTitle":"Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-19","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-20T04:19:11.022Z","createdAt":"2026-11-18T18:48:23.339Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"029806d18c63cad7567a0597f6504f57c0dfe5075c2209fcd35e85e0e0d6a818","kind":"article","url":"https://arxiv.org/abs/2610.26836","sourceUrl":"https://arxiv.org/abs/2610.26836","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-18T18:47:32.263Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.26836","checkedAt":"2026-11-18T19:52:10.547Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"029806d18c63cad7567a0597f6504f57c0dfe5075c2209fcd35e85e0e0d6a818","confidence":0.8400000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"cd32b8448ff55a63b3ad6c7902ed7faf68fdfc5a4843a59ad5b5a756912a0edb","decidedAt":"2026-11-18T18:47:32.888Z"},"provenance":{"textHash":"65a26635581c3b61b8983ea6ff388846c03fc2483e3936d9936d4a6695636b62","observationId":"38c470a91256bdcf2672a522b394738be0ced851e7f9362c38bd2705ee398a6b","retrievedAt":"2026-11-18T18:47:32.182Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"23455e21-ccd4-4464-8566-c069896ae7de"},"reviewer":{"model":"gpt-6-luna","runId":"e4ab5023-d009-4fb5-8d87-3ab4b39d8434"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-18T18:48:23.339Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Zhang, Ke","url":null,"kind":"unknown"},{"name":"Chundur, Sahchit","url":null,"kind":"unknown"},{"name":"Owusu, Jerry Peprah","url":null,"kind":"unknown"},{"name":"Qomi, Mohammad Javad Abdolhosseini","url":null,"kind":"unknown"},{"name":"Raissi, Maziar","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Zhang, Ke","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Ke","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.26836","locator":"head/meta[name=author|citation_author]"},"assertionId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:52:10.547Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc","sourceUrl":"https://arxiv.org/abs/2610.26836","contentSha256":"630bd24732d2a21fd6126c1f57719e04500c58ec2627388eff1273859bc993c8","textHash":"65a26635581c3b61b8983ea6ff388846c03fc2483e3936d9936d4a6695636b62","observedAt":"2026-11-18T19:52:10.547Z"}},{"actor":{"name":"Chundur, Sahchit","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chundur, Sahchit","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.26836","locator":"head/meta[name=author|citation_author]"},"assertionId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:52:10.547Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc","sourceUrl":"https://arxiv.org/abs/2610.26836","contentSha256":"630bd24732d2a21fd6126c1f57719e04500c58ec2627388eff1273859bc993c8","textHash":"65a26635581c3b61b8983ea6ff388846c03fc2483e3936d9936d4a6695636b62","observedAt":"2026-11-18T19:52:10.547Z"}},{"actor":{"name":"Owusu, Jerry Peprah","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Owusu, Jerry Peprah","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.26836","locator":"head/meta[name=author|citation_author]"},"assertionId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:52:10.547Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc","sourceUrl":"https://arxiv.org/abs/2610.26836","contentSha256":"630bd24732d2a21fd6126c1f57719e04500c58ec2627388eff1273859bc993c8","textHash":"65a26635581c3b61b8983ea6ff388846c03fc2483e3936d9936d4a6695636b62","observedAt":"2026-11-18T19:52:10.547Z"}},{"actor":{"name":"Qomi, Mohammad Javad Abdolhosseini","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Qomi, Mohammad Javad Abdolhosseini","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.26836","locator":"head/meta[name=author|citation_author]"},"assertionId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:52:10.547Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc","sourceUrl":"https://arxiv.org/abs/2610.26836","contentSha256":"630bd24732d2a21fd6126c1f57719e04500c58ec2627388eff1273859bc993c8","textHash":"65a26635581c3b61b8983ea6ff388846c03fc2483e3936d9936d4a6695636b62","observedAt":"2026-11-18T19:52:10.547Z"}},{"actor":{"name":"Raissi, Maziar","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Raissi, Maziar","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.26836","locator":"head/meta[name=author|citation_author]"},"assertionId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:52:10.547Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc","sourceUrl":"https://arxiv.org/abs/2610.26836","contentSha256":"630bd24732d2a21fd6126c1f57719e04500c58ec2627388eff1273859bc993c8","textHash":"65a26635581c3b61b8983ea6ff388846c03fc2483e3936d9936d4a6695636b62","observedAt":"2026-11-18T19:52:10.547Z"}}],"participationCoverage":{"total":5,"returned":5,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.26836","snapshotId":"c78c2651504ba0c55aec9a2ca6f77d50ea054454b1d2595904440be643e514bc","contentSha256":"630bd24732d2a21fd6126c1f57719e04500c58ec2627388eff1273859bc993c8","textHash":"65a26635581c3b61b8983ea6ff388846c03fc2483e3936d9936d4a6695636b62","observedAt":"2026-11-18T19:52:10.547Z","basis":"publisher-page-metadata","sourceLinks":45,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/c69e72e9e308f1f46f5dafa9cfdf243439fde68e8f6557d6faf508b4693e754f/source"}}}],"audits":[{"publicationId":"b0079431-068e-4ea2-8ae0-47852db683b2","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the work to Ke Zhang and coauthors on arXiv cs.AI. It keeps the findings and recommendation framed as the authors' reports and argument rather than established fact.","excerpt":"Authors:Ke Zhang, Sahchit Chundur, Jerry Peprah Owusu, Mohammad Javad Abdolhosseini Qomi, Maziar Raissi"},{"dimension":"summary","status":"uncertain","explanation":"The summary and claims refer to 'the three GPT models', but only GPT-5.1 and GPT-5.2 are introduced. GPT-5.4 is missing, so readers cannot tell which third model is meant. The reading also omits the GPT-5.4 nuance: more frequent correct simulations masked an evidence-use loss of about ten points, leaving a net change indistinguishable from zero. Smallest repair: add one sentence on the GPT-5.4 result. 'Context window' also appears without a plain-language gloss.","excerpt":"For GPT-5.4, more frequent correct simulations mask an evidence-use loss of about ten points, leaving a net change indistinguishable from zero."},{"dimension":"date","status":"pass","explanation":"originalPublished is 2026-10-19, which matches the v1 submission date. It does not use the 16 Nov 2026 revision or the feed date.","excerpt":"Submitted on 19 Oct 2026 (v1), last revised 16 Nov 2026 (this version, v2)"},{"dimension":"support","status":"uncertain","explanation":"The abstract says that, for GPT-5.1 and GPT-5.2, inlining outputs 'works as well' (as naming the authoritative output), 'while asking for correct rounding does not.' This most naturally means rounding requests did not work as well as naming. The reading's claim that they 'did not improve results' may overstate this. Smallest repair: say asking for correct rounding did not work as well. The other claims are supported, and the limitations are appropriately framed.","excerpt":"for GPT-5.1 and GPT-5.2, inlining outputs that fit works as well, while asking for correct rounding does not."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the arXiv abstract page, which is the original source.","excerpt":null}],"createdAt":"2026-11-18T18:48:23.339Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}