# Represented but Not Used: Value-Policy Dissociation in LLM Agents — arXiv cs.AI [Canonical reading](https://escapemeridian.com/readings/represented-but-not-used-value-policy-dissociation-in-llm-a53cb7ad) Yuzhang Luo, Mohamed Aghzal, Liangming Pan and Ziyu Yao argue that an LLM agent can represent which action is more valuable without choosing it. In reinforcement learning, action value describes how an action affects future outcomes. The authors study this gap, which they call value-policy dissociation, in three models and two settings: a partially observed Grid environment and WebShop, a simulated web shopping environment. They report that relative action values can be decoded from intermediate hidden states even when a model’s policy prefers a worse action. But directly intervening along the value readout direction has limited and inconsistent effects. Natural Value Transport (NVT), a transformation learned from hidden state changes associated with differences in action value, more consistently influences action preferences, particularly at later layers. The authors also report improved task outcomes with a closed-loop controller that uses the value readout as a sensor and NVT as an actuator. Their results suggest a distinction between representing prospective value and using it to select actions: finding useful information inside a model does not, by itself, show that its policy will act on it. The evidence here is from the abstract; it does not give the size of the reported improvements or establish how broadly they apply beyond the tested settings. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Nov 7, 2026 (2026-11-07). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-10T15:47:25.982Z. First observed: 2026-11-10T08:34:37.240Z. Retrieved: 2026-11-10T15:33:58.399Z. - [Represented but Not Used: Value-Policy Dissociation in LLM Agents](https://arxiv.org/abs/2611.10457) ## Limits The supplied evidence is the abstract, which does not give the size of the reported task-outcome improvements. The abstract reports findings in a partially observed Grid environment and WebShop; it does not establish how broadly they apply beyond those settings. ## Open question How much do the reported effects of NVT and the closed-loop controller vary across the tested models and environments? ## Source-stated The authors define value-policy dissociation as a gap between representing action values and the policy’s choice. > We call this gap between value representation and policy choice value-policy dissociation. - [Supporting source](https://arxiv.org/abs/2611.10457) ## Source-stated The authors report that relative action values are linearly decodable from intermediate hidden states, even when the policy prefers a worse action. > the relative values of candidate actions are linearly decodable from intermediate hidden states, including at states where the policy prefers a worse action. - [Supporting source](https://arxiv.org/abs/2611.10457) ## Source-stated The authors report that NVT influences action preferences more consistently than interventions along the value readout direction, particularly at later layers. > While readout-direction interventions have limited and inconsistent effects, NVT more consistently influences action preferences, particularly at later layers. - [Supporting source](https://arxiv.org/abs/2611.10457) ## Source-stated The authors report improved task outcomes from a closed-loop controller using the value readout as a sensor and NVT as an actuator. > We also test a closed-loop controller that uses the value readout as a sensor and NVT as an actuator, observing improvements in task outcomes. - [Supporting source](https://arxiv.org/abs/2611.10457) ## Source-stated The results suggest that representing prospective value does not ensure that an agent’s policy will use it when choosing actions. > These results suggest that LLM agents can represent useful prospective value without reliably translating it into policy choice. - [Supporting source](https://arxiv.org/abs/2611.10457) ## Source and contributor context Luo, Yuzhang, Aghzal, Mohamed, Pan, Liangming, Yao, Ziyu · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Reinforcement learning Training by trial and error, in which a system is rewarded for the outcomes its designers want. - [Definition reference](https://developers.google.com/machine-learning/glossary#reinforcement-learning-rl) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 5518670b-06fa-417f-862a-030233640ad4. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-10T15:47:25.982Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 22953bd6b00a254b093b9d0d607ac65f4fbe8181a9e0fccdd7fa511643856025. Source text hash: deaff12172a64b5ed84b7cc6545f5f453afb0146b6d41342d43c03fa3c87d3c1. ## Revision and source-check history ### Version 1 — 2026-11-10T15:47:25.982Z Yuzhang Luo, Mohamed Aghzal, Liangming Pan and Ziyu Yao argue that an LLM agent can represent which action is more valuable without choosing it. In reinforcement learning, action value describes how an action affects future outcomes. The authors study this gap, which they call value-policy dissociation, in three models and two settings: a partially observed Grid environment and WebShop, a simulated web shopping environment. They report that relative action values can be decoded from intermediate hidden states even when a model’s policy prefers a worse action. But directly intervening along the value readout direction has limited and inconsistent effects. Natural Value Transport (NVT), a transformation learned from hidden state changes associated with differences in action value, more consistently influences action preferences, particularly at later layers. The authors also report improved task outcomes with a closed-loop controller that uses the value readout as a sensor and NVT as an actuator. Their results suggest a distinction between representing prospective value and using it to select actions: finding useful information inside a model does not, by itself, show that its policy will act on it. The evidence here is from the abstract; it does not give the size of the reported improvements or establish how broadly they apply beyond the tested settings. ## Original source arXiv cs.AI · Article Original source date: Nov 7, 2026 (2026-11-07). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-10T15:47:25.982Z. First observed: 2026-11-10T08:34:37.240Z. Retrieved: 2026-11-10T15:33:58.399Z. - [Represented but Not Used: Value-Policy Dissociation in LLM Agents](https://arxiv.org/abs/2611.10457) ## Limits The supplied evidence is the abstract, which does not give the size of the reported task-outcome improvements. The abstract reports findings in a partially observed Grid environment and WebShop; it does not establish how broadly they apply beyond those settings. ## Open question How much do the reported effects of NVT and the closed-loop controller vary across the tested models and environments? ## Source-stated The authors define value-policy dissociation as a gap between representing action values and the policy’s choice. > We call this gap between value representation and policy choice value-policy dissociation. - [Supporting source](https://arxiv.org/abs/2611.10457) ## Source-stated The authors report that relative action values are linearly decodable from intermediate hidden states, even when the policy prefers a worse action. > the relative values of candidate actions are linearly decodable from intermediate hidden states, including at states where the policy prefers a worse action. - [Supporting source](https://arxiv.org/abs/2611.10457) ## Source-stated The authors report that NVT influences action preferences more consistently than interventions along the value readout direction, particularly at later layers. > While readout-direction interventions have limited and inconsistent effects, NVT more consistently influences action preferences, particularly at later layers. - [Supporting source](https://arxiv.org/abs/2611.10457) ## Source-stated The authors report improved task outcomes from a closed-loop controller using the value readout as a sensor and NVT as an actuator. > We also test a closed-loop controller that uses the value readout as a sensor and NVT as an actuator, observing improvements in task outcomes. - [Supporting source](https://arxiv.org/abs/2611.10457) ## Source-stated The results suggest that representing prospective value does not ensure that an agent’s policy will use it when choosing actions. > These results suggest that LLM agents can represent useful prospective value without reliably translating it into policy choice. - [Supporting source](https://arxiv.org/abs/2611.10457) ## Source and contributor context Luo, Yuzhang, Aghzal, Mohamed, Pan, Liangming, Yao, Ziyu · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Reinforcement learning Training by trial and error, in which a system is rewarded for the outcomes its designers want. - [Definition reference](https://developers.google.com/machine-learning/glossary#reinforcement-learning-rl) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 5518670b-06fa-417f-862a-030233640ad4. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-10T15:47:25.982Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 22953bd6b00a254b093b9d0d607ac65f4fbe8181a9e0fccdd7fa511643856025. Source text hash: deaff12172a64b5ed84b7cc6545f5f453afb0146b6d41342d43c03fa3c87d3c1. ### passed — 2026-11-10T15:47:25.982Z — anthropic/claude-opus-5.5 Publication: 5518670b-06fa-417f-862a-030233640ad4 attribution — pass: The reading credits the four listed authors and the arXiv cs.AI preprint correctly. Findings are framed as what the authors report or suggest, not as established fact. > Authors:Yuzhang Luo, Mohamed Aghzal, Liangming Pan, Ziyu Yao summary — pass: The summary leads with the central thesis and explains action value in plain language. It covers both test environments, the two intervention approaches, the controller, and the main conclusion. The question about variation across models and settings follows from the evidence. One small repair: the summary says values 'can be decoded', while the source says 'linearly decodable'. Restoring 'linearly' would keep the technical term exact. Terms like hidden states and policy rely on tooltips, which is acceptable. > These results suggest that LLM agents can represent useful prospective value without reliably translating it into policy choice. date — pass: originalPublished is 2026-11-07, which matches the v1 submission date on the arXiv page. The later feed date of 2026-11-10 is correctly not used. > \[Submitted on 7 Nov 2026\] support — pass: Every claim is backed by a verbatim abstract excerpt and kept attributed. The comparison of interventions and the controller result are not overstated. The stated limitations correctly note that the abstract gives no effect sizes and does not establish generality beyond the Grid and WebShop settings. The reading invents no beneficiaries, costs or forecasts. > While readout-direction interventions have limited and inconsistent effects, NVT more consistently influences action preferences, particularly at later layers. media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/a53cb7adcb65f7a2e5d3883909e01316525e083a8207a14a8e4116ef608d730e/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/represented-but-not-used-value-policy-dissociation-in-llm-a53cb7ad.json)