Tang, Yuhan, Fan, Ke, Zou, Xinkai, Mo, Baichuan, Cui, Kangxin, Qu, Ao, Zheng, Ou, Yu, Jiangbo, Koutsopoulos, Haris N., Zhao, Jinhua, Zhou, Yuan · Nov 9, 2026
From Questions to Decisions: Benchmarking How LLM Agents Acquire Decision-Relevant Information from Human Beings
Markdown · JSON · Original source
The authors of Ask2Decide argue that agents still struggle to ask people for information that changes a plan. Their tests 43 planning tasks: an operator holds facts missing from the request and records, while optimization solvers identify which facts change the optimal plan. Similar-sounding facts that do not change the plan test whether agents can distinguish useful information from irrelevant details. In 79 interaction sessions with 19 participants, human replies after an incorrect plan omitted the correcting fact more often than replies from an uncalibrated simulated user: 43% versus 7%. The authors use those interactions to calibrate a simulated operator. Across four frontier LLMs, initial plans were infeasible on 58–98% of tasks when information was missing, versus 5–28% when all facts were supplied upfront. Agents uncovered facts that changed the optimal plan only slightly more often than facts that left it unchanged. The authors argue that this makes obtaining decision-relevant information through interaction a major bottleneck for LLM planning. The paper describes the work as ongoing and says it will be updated.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 9, 2026 (2026-11-09).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-12T00:17:45.770Z. First observed: 2026-11-11T05:06:49.684Z. Retrieved: 2026-11-12T00:16:42.470Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The reported evaluation covers 43 planning tasks, 79 interaction sessions with 19 participants, and four frontier ; the abstract does not establish how well the results generalize beyond those settings. The authors describe the work as ongoing and say it will be updated.
Open question
As this ongoing work is updated, do agents become better at uncovering facts that change the optimal plan than facts that leave it unchanged?
Source-stated
The authors introduce Ask2Decide, a benchmark testing whether agents ask for missing information that affects planning decisions.
We introduce Ask2Decide, a benchmark of 43 planning tasks that evaluates whether agents ask for missing information that matters for their decisions.
Source-stated
In the reported participant sessions, human replies after an incorrect plan omitted the correcting fact more often than replies from an uncalibrated simulated user.
human replies omit the correcting fact far more often than replies from an uncalibrated simulated user (43% versus 7%).
Source-stated
Across four evaluated frontier LLMs, initial plans were more often infeasible when information was missing than when all facts were provided upfront.
initial plans are infeasible on 58-98% of tasks when information is missing, compared with 5-28% when all facts are provided upfront.
Source-stated
The authors report that agents only slightly more often uncovered facts that changed the optimal plan than facts that did not change it.
Agents uncover facts that change the optimal plan only slightly more often than facts that leave it unchanged (39-52% versus 32-41%).
Source-stated
The authors characterize obtaining decision-relevant information through interaction as a major bottleneck for LLM planning.
These results show that obtaining decision-relevant information through interaction remains a major bottleneck for LLM planning.
Source and contributor context
Tang, Yuhan, Fan, Ke, Zou, Xinkai, Mo, Baichuan, Cui, Kangxin, Qu, Ao, Zheng, Ou, Yu, Jiangbo, Koutsopoulos, Haris N., Zhao, Jinhua, Zhou, Yuan · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 4fb012fa-ee07-4489-8357-b1d80de3b3e3.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T00:17:45.770Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: b24f2a1ea101f5ac270fc98f1550fb0f0a4313b3f94275e9543cd2ca4beefec5. Source text hash: e9c702526714e7e161ed4ee593ca6106e0f3b14abca4797c5ad31fe2f8646dc3.
Revision and source-check history
Version 1 · 2026-11-12T00:17:45.770Z
From Questions to Decisions: Benchmarking How LLM Agents Acquire Decision-Relevant Information from Human Beings
The authors of Ask2Decide argue that agents still struggle to ask people for information that changes a plan. Their tests 43 planning tasks: an operator holds facts missing from the request and records, while optimization solvers identify which facts change the optimal plan. Similar-sounding facts that do not change the plan test whether agents can distinguish useful information from irrelevant details. In 79 interaction sessions with 19 participants, human replies after an incorrect plan omitted the correcting fact more often than replies from an uncalibrated simulated user: 43% versus 7%. The authors use those interactions to calibrate a simulated operator. Across four frontier LLMs, initial plans were infeasible on 58–98% of tasks when information was missing, versus 5–28% when all facts were supplied upfront. Agents uncovered facts that changed the optimal plan only slightly more often than facts that left it unchanged. The authors argue that this makes obtaining decision-relevant information through interaction a major bottleneck for LLM planning. The paper describes the work as ongoing and says it will be updated.
Original source
cs.AI · Article
Source details
Original source date: Nov 9, 2026 (2026-11-09).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-12T00:17:45.770Z. First observed: 2026-11-11T05:06:49.684Z. Retrieved: 2026-11-12T00:16:42.470Z.
Limits
The reported evaluation covers 43 planning tasks, 79 interaction sessions with 19 participants, and four frontier ; the abstract does not establish how well the results generalize beyond those settings. The authors describe the work as ongoing and say it will be updated.
Open question
As this ongoing work is updated, do agents become better at uncovering facts that change the optimal plan than facts that leave it unchanged?
Source-stated
The authors introduce Ask2Decide, a benchmark testing whether agents ask for missing information that affects planning decisions.
We introduce Ask2Decide, a benchmark of 43 planning tasks that evaluates whether agents ask for missing information that matters for their decisions.
Source-stated
In the reported participant sessions, human replies after an incorrect plan omitted the correcting fact more often than replies from an uncalibrated simulated user.
human replies omit the correcting fact far more often than replies from an uncalibrated simulated user (43% versus 7%).
Source-stated
Across four evaluated frontier LLMs, initial plans were more often infeasible when information was missing than when all facts were provided upfront.
initial plans are infeasible on 58-98% of tasks when information is missing, compared with 5-28% when all facts are provided upfront.
Source-stated
The authors report that agents only slightly more often uncovered facts that changed the optimal plan than facts that did not change it.
Agents uncover facts that change the optimal plan only slightly more often than facts that leave it unchanged (39-52% versus 32-41%).
Source-stated
The authors characterize obtaining decision-relevant information through interaction as a major bottleneck for LLM planning.
These results show that obtaining decision-relevant information through interaction remains a major bottleneck for LLM planning.
Source and contributor context
Tang, Yuhan, Fan, Ke, Zou, Xinkai, Mo, Baichuan, Cui, Kangxin, Qu, Ao, Zheng, Ou, Yu, Jiangbo, Koutsopoulos, Haris N., Zhao, Jinhua, Zhou, Yuan · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 4fb012fa-ee07-4489-8357-b1d80de3b3e3.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T00:17:45.770Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: b24f2a1ea101f5ac270fc98f1550fb0f0a4313b3f94275e9543cd2ca4beefec5. Source text hash: e9c702526714e7e161ed4ee593ca6106e0f3b14abca4797c5ad31fe2f8646dc3.
uncertain · 2026-11-12T00:17:45.770Z · anthropic/claude-opus-5.5
Publication: 4fb012fa-ee07-4489-8357-b1d80de3b3e3
attribution — pass: The findings and the bottleneck conclusion are consistently attributed to the paper's authors and presented as their reported results, not as established fact. The arXiv author list matches the source.
Authors:Yuhan Tang, Ke Fan, Xinkai Zou, Baichuan Mo, Kangxin Cui, Ao Qu, Ou Zheng, Jiangbo Yu, Haris N. Koutsopoulos, Jinhua Zhao, Yuan Zhou
summary — uncertain: The summary is accurate but omits context a lay reader needs. It does not say what these agents do: the abstract's examples are scheduling drivers and assigning nurses, where requests are incomplete or outdated. It also drops the authors' reason the benchmark exists: earlier tests used fully specified problems or uncalibrated simulated users. Without that, the 43% versus 7% result reads as a curiosity, not evidence that simulated users are too forthcoming. 'Far more often' is weakened to 'more often', and 'infeasible' and 'operator' are unexplained. Smallest repair: add one sentence with the use-case examples and the gap in existing benchmarks.
Existing benchmarks typically provide fully specified problems or rely on uncalibrated LLM-simulated users, leaving this ability insufficiently tested.
date — pass: The originalPublished value of 2026-11-09 matches the v1 submission date on arXiv. The later feed date is not the original publication date.
[Submitted on 9 Nov 2026]
support — pass: Every claim and number matches the abstract verbatim: 43 tasks, 79 sessions, 19 participants, 43% versus 7%, 58-98% versus 5-28%, and slightly more often. The limitations and the question follow from the source without adding benefits or costs. The question is grounded in the authors' note that the work is ongoing and will be updated.
Agents uncover facts that change the optimal plan only slightly more often than facts that leave it unchanged (39-52% versus 32-41%).
media — pass: The selected media is the verified direct link to the arXiv abstract page. It is the original source and the most relevant choice.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A language model with many learned numerical settings, used to process and generate language.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition reference