# arXiv cs.AI: Thinking Harder Raises Confidence, Not Correctness in LLM Shopping Agents

[Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-thinking-harder-raises-confidence-not-correctness-04539dc5)

Authors Adi Krishnamoorthy and X.Y. Han report that more thinking effort did not improve LLM shopping agents’ decision quality in a statistically meaningful way across three product-choice benchmarks, but significantly raised their stated confidence. They tested ten frontier models on benchmarks built from WebShop, Amazon’s SQID and Wayfair’s WANDS. The authors also report that most models were not well-calibrated at every effort level. The result challenges the assumption that asking a shopping agent to think harder will make its recommendations better. Escape Meridian interpretation: if agents’ confidence rises without improved choices, confidence alone may not help consumers judge whether a recommendation is reliable. The authors further report shared errors: several models tended to err on the same tasks, and many chose the same wrong product on tasks where several models erred. The abstract does not establish why these errors overlap or how the findings apply beyond the tested benchmarks. It reports benchmark results, not evidence about consumers’ real-world purchasing outcomes.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv cs.AI · Article

Original source date: Nov 17, 2026 (2026-11-17).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-18T16:03:45.843Z. First observed: 2026-11-18T05:04:33.280Z. Retrieved: 2026-11-18T16:02:55.334Z.

- [Thinking Harder Doesn't Lead to Better Choices: Reasoning Effort Increases the Confidence but Not the Correctness of LLM Shopping Agents](https://arxiv.org/abs/2611.22033)

## Limits

The evidence provides the paper’s abstract but not detailed methods, benchmark results, or analysis of why models share errors. The reported findings concern three product-choice benchmarks; the abstract does not establish how they apply to consumer purchasing outside those tasks.


## Open question

What accounts for several models choosing the same wrong product on tasks where they err?


## Source-stated

The authors report that increasing thinking effort did not improve decision quality in a statistically meaningful way across the three benchmarks.

> Across all three benchmarks, we found that increasing the thinking effort does not increase decision quality in a statistically meaningful way.

- [Supporting source](https://arxiv.org/abs/2611.22033)

## Source-stated

The authors report that more thinking effort significantly raised stated confidence.

> The same change in thinking effort raises stated confidence at a statistically significant level, so extra thinking buys confidence rather than correctness.

- [Supporting source](https://arxiv.org/abs/2611.22033)

## Source-stated

The authors tested ten frontier models on three product-choice benchmarks built from WebShop, Amazon's SQID and Wayfair's WANDS.

> We benchmark ten frontier models on three product-choice benchmarks built from WebShop, Amazon's SQID and Wayfair's WANDS platforms

- [Supporting source](https://arxiv.org/abs/2611.22033)

## Source-stated

The authors report that most models were not well-calibrated at every effort level.

> most models are not well-calibrated at every effort level.

- [Supporting source](https://arxiv.org/abs/2611.22033)

## Source-stated

The authors report that models often shared errors, and that many selected the same wrong product when several models erred on a task.

> when one model errs in a task, several others tend to err on that same task. Furthermore, on tasks where several models all err, a large number of them pick the same wrong product.

- [Supporting source](https://arxiv.org/abs/2611.22033)

## Escape Meridian interpretation

The abstract describes benchmark results, not observed consumer purchasing outcomes.

> We benchmark ten frontier models on three product-choice benchmarks built from WebShop, Amazon's SQID and Wayfair's WANDS platforms

- [Supporting source](https://arxiv.org/abs/2611.22033)

## Source and contributor context

Krishnamoorthy, Adi, Han, X. Y. · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

- [Definition reference](https://arxiv.org/abs/2211.09110)

## Frontier model

One of the most capable AI models available at a given time; the set shifts as models improve.

- [Definition reference](https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper)

## Model calibration

Whether a model's stated confidence matches how often it turns out to be right.

- [Definition reference](https://en.wikipedia.org/wiki/Calibration_%28statistics%29)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: d1aa6121-1daf-43ff-831e-cec38552b179.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T16:03:45.843Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 49ca5deac8b8fefe44bfaacf2a8dc71bcb751efc50520b738f71c0aca04c5907. Source text hash: 46efce1a5b36e1a5e6895721af01785e3e430f3048f5e5604b1e804d1c6ec9f0.


## Revision and source-check history

### Version 1 — 2026-11-18T16:03:45.843Z

Authors Adi Krishnamoorthy and X.Y. Han report that more thinking effort did not improve LLM shopping agents’ decision quality in a statistically meaningful way across three product-choice benchmarks, but significantly raised their stated confidence. They tested ten frontier models on benchmarks built from WebShop, Amazon’s SQID and Wayfair’s WANDS. The authors also report that most models were not well-calibrated at every effort level. The result challenges the assumption that asking a shopping agent to think harder will make its recommendations better. Escape Meridian interpretation: if agents’ confidence rises without improved choices, confidence alone may not help consumers judge whether a recommendation is reliable. The authors further report shared errors: several models tended to err on the same tasks, and many chose the same wrong product on tasks where several models erred. The abstract does not establish why these errors overlap or how the findings apply beyond the tested benchmarks. It reports benchmark results, not evidence about consumers’ real-world purchasing outcomes.

## Original source

arXiv cs.AI · Article

Original source date: Nov 17, 2026 (2026-11-17).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-18T16:03:45.843Z. First observed: 2026-11-18T05:04:33.280Z. Retrieved: 2026-11-18T16:02:55.334Z.

- [Thinking Harder Doesn't Lead to Better Choices: Reasoning Effort Increases the Confidence but Not the Correctness of LLM Shopping Agents](https://arxiv.org/abs/2611.22033)

## Limits

The evidence provides the paper’s abstract but not detailed methods, benchmark results, or analysis of why models share errors. The reported findings concern three product-choice benchmarks; the abstract does not establish how they apply to consumer purchasing outside those tasks.


## Open question

What accounts for several models choosing the same wrong product on tasks where they err?


## Source-stated

The authors report that increasing thinking effort did not improve decision quality in a statistically meaningful way across the three benchmarks.

> Across all three benchmarks, we found that increasing the thinking effort does not increase decision quality in a statistically meaningful way.

- [Supporting source](https://arxiv.org/abs/2611.22033)

## Source-stated

The authors report that more thinking effort significantly raised stated confidence.

> The same change in thinking effort raises stated confidence at a statistically significant level, so extra thinking buys confidence rather than correctness.

- [Supporting source](https://arxiv.org/abs/2611.22033)

## Source-stated

The authors tested ten frontier models on three product-choice benchmarks built from WebShop, Amazon's SQID and Wayfair's WANDS.

> We benchmark ten frontier models on three product-choice benchmarks built from WebShop, Amazon's SQID and Wayfair's WANDS platforms

- [Supporting source](https://arxiv.org/abs/2611.22033)

## Source-stated

The authors report that most models were not well-calibrated at every effort level.

> most models are not well-calibrated at every effort level.

- [Supporting source](https://arxiv.org/abs/2611.22033)

## Source-stated

The authors report that models often shared errors, and that many selected the same wrong product when several models erred on a task.

> when one model errs in a task, several others tend to err on that same task. Furthermore, on tasks where several models all err, a large number of them pick the same wrong product.

- [Supporting source](https://arxiv.org/abs/2611.22033)

## Escape Meridian interpretation

The abstract describes benchmark results, not observed consumer purchasing outcomes.

> We benchmark ten frontier models on three product-choice benchmarks built from WebShop, Amazon's SQID and Wayfair's WANDS platforms

- [Supporting source](https://arxiv.org/abs/2611.22033)

## Source and contributor context

Krishnamoorthy, Adi, Han, X. Y. · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

- [Definition reference](https://arxiv.org/abs/2211.09110)

## Frontier model

One of the most capable AI models available at a given time; the set shifts as models improve.

- [Definition reference](https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper)

## Model calibration

Whether a model's stated confidence matches how often it turns out to be right.

- [Definition reference](https://en.wikipedia.org/wiki/Calibration_%28statistics%29)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: d1aa6121-1daf-43ff-831e-cec38552b179.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T16:03:45.843Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 49ca5deac8b8fefe44bfaacf2a8dc71bcb751efc50520b738f71c0aca04c5907. Source text hash: 46efce1a5b36e1a5e6895721af01785e3e430f3048f5e5604b1e804d1c6ec9f0.


### passed — 2026-11-18T16:03:45.843Z — anthropic/claude-opus-5.5

Publication: d1aa6121-1daf-43ff-831e-cec38552b179

attribution — pass: The reading correctly attributes the findings to the paper's listed authors, Adi Krishnamoorthy and X.Y. Han, on arXiv cs.AI. It keeps results framed as what the authors report. The submitter field reads 'Aditya Krishnamoorthy', but the author line matches the draft, so this does not affect attribution.

> Authors:Adi Krishnamoorthy, X.Y. Han

summary — pass: The reading opens with the central claim: more thinking effort raised stated confidence but not decision quality. It names the tested setup and keeps the authors' framing as a test of a common assumption. It also covers the shared-error finding and the limits of the evidence. 'Calibrated' is not explained in plain words, though the source defines it (reported confidence vs. actual correctness). It is a secondary finding and can be handled by a tooltip, so it does not block understanding. The question follows from the evidence without leading.

> so extra thinking buys confidence rather than correctness

date — pass: The originalPublished date of 2026-11-17 matches the arXiv v1 submission date. The feed date of 2026-11-18 is correctly not used.

> \[Submitted on 17 Nov 2026\]

support — pass: Every factual claim matches the abstract word for word or nearly so. The Escape Meridian interpretation, that confidence alone may not help consumers judge reliability, is labeled, conditional and grounded in the confidence-versus-correctness finding. The draft also states that the evidence covers benchmarks, not real-world purchasing outcomes, and that the cause of the shared errors is unknown. No unsupported benefits or harms are added.

> The same change in thinking effort raises stated confidence at a statistically significant level

media — pass: The selected media is the verified direct link to the original arXiv abstract page. This is the most direct way for readers to reach the source, and no other source media was found.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/04539dc5150fe6a2007c234f271bf9902f76d8c26524b9be89da8ad38deafc45/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-thinking-harder-raises-confidence-not-correctness-04539dc5.json)
