# arXiv cs.AI paper reports that language models often answer underspecified questions without signaling uncertainty

[Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-paper-reports-that-language-models-often-answer-cae90331)

In “Wrong with Confidence,” Md. Rejaul Korim Sadi, Toufiqur Rahman Tasin, Golam Mostofa Naeem, Dewan Fahad Chowdhury and Hatim Al Amin Chowdhury report a paired study of five open-weight models (models whose weights are openly available). They compared answers to factual questions with and without a needed referent (the specific thing or person a question refers to), using the same answer key. The models asked for clarification on 33.7% of underspecified prompts and answered the rest; among attempted answers, errors rose from 39.1% on specified questions to 52.0% on underspecified ones. The authors say verbal hedging was a weak signal of correctness: 75.3% of wrong answers had no hedge. Their point is not simply that models can be wrong, but that a confident-sounding guess may not tell users when essential information is missing. They argue evaluations should count clarification separately from answers and check label validity as well as rater agreement. The study also reports that models used gendered pronouns following occupational stereotypes when they chose a pronoun for an occupation, longer answers when a user identity was stated, and no reduction in safety caveats when users claimed expertise. These findings concern the five models and study prompts tested; the abstract does not establish how broadly the patterns apply beyond them.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv cs.AI · Article

Original source date: Nov 9, 2026 (2026-11-09).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-17T10:33:52.659Z. First observed: 2026-11-17T08:31:07.938Z. Retrieved: 2026-11-17T10:32:25.802Z.

- [Wrong with Confidence: Measuring Hallucination, Overconfidence and Identity-Conditioned Behaviour in Large Language Models](https://arxiv.org/abs/2611.18728)

## Limits

The study tests five open-weight models and 142 factual questions; the abstract does not establish whether its findings generalize beyond these models and prompts. The abstract reports two raters and mixed-effects models but does not provide detailed results on rater agreement or label validity.


## Open question

Do the reported patterns hold beyond these five open-weight models and the study’s 142 factual questions?


## Source-stated

The authors compared five open-weight models on paired factual questions, with and without a referent, using the same answer key.

> Each of 142 factual questions was posed in a fully specified form and in a referentially underspecified form with the same answer key

- [Supporting source](https://arxiv.org/abs/2611.18728)

## Source-stated

The models asked for clarification on 33.7% of underspecified prompts; among attempted answers, the error rate rose from 39.1% to 52.0%.

> Without the referent, the models sought clarification on 33.7% of prompts and answered the rest. Among attempted answers, the error rate rose from 39.1% to 52.0%

- [Supporting source](https://arxiv.org/abs/2611.18728)

## Source-stated

The authors report that verbal hedging did not reliably indicate whether an answer was wrong.

> 75.3% of wrong answers carried no hedge, wrong answers to underspecified prompts were hedged no more often than wrong answers to clear prompts (27.3% vs 22.3%), and hedging separated correct from incorrect answers only weakly (AUROC 0.56).

- [Supporting source](https://arxiv.org/abs/2611.18728)

## Source-stated

The authors argue that evaluations should count clarification separately and check label validity as well as rater agreement.

> Evaluations should score clarification as a distinct outcome and check label validity as well as rater agreement.

- [Supporting source](https://arxiv.org/abs/2611.18728)

## Source-stated

The reported behavioral patterns include stereotyped gendered pronouns, longer answers when user identity was stated, and unchanged safety caveats when users claimed expertise.

> Gendered pronouns followed occupational stereotypes whenever a model committed to one; any stated user identity lengthened answers by about 45% relative to an anonymous user; claimed expertise did not reduce safety caveats.

- [Supporting source](https://arxiv.org/abs/2611.18728)

## Source and contributor context

Sadi, Md. Rejaul Korim, Tasin, Toufiqur Rahman, Naeem, Golam Mostofa, Chowdhury, Dewan Fahad, Chowdhury, Hatim Al Amin · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Open weights

A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.

- [Definition reference](https://opensource.org/ai/open-weights)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: 557e8f56-549b-4cad-86d2-6a12092c5e2f.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-17T10:33:52.659Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: a748977080f01b1d92cfe2ba3dbf923859a5661cf158ba7be19b225c1de85bc0. Source text hash: 57eee0d00e108529540388a1a07f25e58a4043dfca4b2d092cbfcb68de6d1540.

Correction reason: Clarified the meanings of “referent” and “open-weight” for general readers and repaired the occupational-pronoun wording to specify the reported condition.


## Revision and source-check history

### Version 1 — 2026-11-17T10:33:52.659Z

In “Wrong with Confidence,” Md. Rejaul Korim Sadi, Toufiqur Rahman Tasin, Golam Mostofa Naeem, Dewan Fahad Chowdhury and Hatim Al Amin Chowdhury report a paired study of five open-weight models (models whose weights are openly available). They compared answers to factual questions with and without a needed referent (the specific thing or person a question refers to), using the same answer key. The models asked for clarification on 33.7% of underspecified prompts and answered the rest; among attempted answers, errors rose from 39.1% on specified questions to 52.0% on underspecified ones. The authors say verbal hedging was a weak signal of correctness: 75.3% of wrong answers had no hedge. Their point is not simply that models can be wrong, but that a confident-sounding guess may not tell users when essential information is missing. They argue evaluations should count clarification separately from answers and check label validity as well as rater agreement. The study also reports that models used gendered pronouns following occupational stereotypes when they chose a pronoun for an occupation, longer answers when a user identity was stated, and no reduction in safety caveats when users claimed expertise. These findings concern the five models and study prompts tested; the abstract does not establish how broadly the patterns apply beyond them.

## Original source

arXiv cs.AI · Article

Original source date: Nov 9, 2026 (2026-11-09).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-17T10:33:52.659Z. First observed: 2026-11-17T08:31:07.938Z. Retrieved: 2026-11-17T10:32:25.802Z.

- [Wrong with Confidence: Measuring Hallucination, Overconfidence and Identity-Conditioned Behaviour in Large Language Models](https://arxiv.org/abs/2611.18728)

## Limits

The study tests five open-weight models and 142 factual questions; the abstract does not establish whether its findings generalize beyond these models and prompts. The abstract reports two raters and mixed-effects models but does not provide detailed results on rater agreement or label validity.


## Open question

Do the reported patterns hold beyond these five open-weight models and the study’s 142 factual questions?


## Source-stated

The authors compared five open-weight models on paired factual questions, with and without a referent, using the same answer key.

> Each of 142 factual questions was posed in a fully specified form and in a referentially underspecified form with the same answer key

- [Supporting source](https://arxiv.org/abs/2611.18728)

## Source-stated

The models asked for clarification on 33.7% of underspecified prompts; among attempted answers, the error rate rose from 39.1% to 52.0%.

> Without the referent, the models sought clarification on 33.7% of prompts and answered the rest. Among attempted answers, the error rate rose from 39.1% to 52.0%

- [Supporting source](https://arxiv.org/abs/2611.18728)

## Source-stated

The authors report that verbal hedging did not reliably indicate whether an answer was wrong.

> 75.3% of wrong answers carried no hedge, wrong answers to underspecified prompts were hedged no more often than wrong answers to clear prompts (27.3% vs 22.3%), and hedging separated correct from incorrect answers only weakly (AUROC 0.56).

- [Supporting source](https://arxiv.org/abs/2611.18728)

## Source-stated

The authors argue that evaluations should count clarification separately and check label validity as well as rater agreement.

> Evaluations should score clarification as a distinct outcome and check label validity as well as rater agreement.

- [Supporting source](https://arxiv.org/abs/2611.18728)

## Source-stated

The reported behavioral patterns include stereotyped gendered pronouns, longer answers when user identity was stated, and unchanged safety caveats when users claimed expertise.

> Gendered pronouns followed occupational stereotypes whenever a model committed to one; any stated user identity lengthened answers by about 45% relative to an anonymous user; claimed expertise did not reduce safety caveats.

- [Supporting source](https://arxiv.org/abs/2611.18728)

## Source and contributor context

Sadi, Md. Rejaul Korim, Tasin, Toufiqur Rahman, Naeem, Golam Mostofa, Chowdhury, Dewan Fahad, Chowdhury, Hatim Al Amin · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Open weights

A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.

- [Definition reference](https://opensource.org/ai/open-weights)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: 557e8f56-549b-4cad-86d2-6a12092c5e2f.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-17T10:33:52.659Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: a748977080f01b1d92cfe2ba3dbf923859a5661cf158ba7be19b225c1de85bc0. Source text hash: 57eee0d00e108529540388a1a07f25e58a4043dfca4b2d092cbfcb68de6d1540.

Correction reason: Clarified the meanings of “referent” and “open-weight” for general readers and repaired the occupational-pronoun wording to specify the reported condition.


### passed — 2026-11-17T10:33:52.659Z — anthropic/claude-opus-5.5

Publication: 557e8f56-549b-4cad-86d2-6a12092c5e2f

attribution — pass: The draft names the five listed authors and attributes the findings and recommendations to them as their report. It does not present them as established fact or as Escape Meridian's view.

> Authors:Md. Rejaul Korim Sadi, Toufiqur Rahman Tasin, Golam Mostofa Naeem, Dewan Fahad Chowdhury, Hatim Al Amin Chowdhury

summary — pass: The draft leads with the actor and the study design, and explains 'open-weight' and 'referent' in plain language. The numbers are accurate, and the error rates are correctly scoped to attempted answers. The central point, that a guess's wording does not reveal it is a guess, preserves the authors' force. The question about generalization is a genuine open issue rather than a leading claim. Calling the paper 'arXiv cs.AI' is acceptable because the paper is cross-listed there, although its primary category is cs.CL.

> The models either ask or answer as if the question were well posed, and the wording of a guess does not reveal that it is one.

date — pass: originalPublished is 2026-11-09, which matches the v1 submission date. It is not the later feed date of 2026-11-17.

> \[Submitted on 9 Nov 2026\]

support — pass: Each claim is backed by a verbatim abstract excerpt. The limitations accurately note the five-model, 142-question scope, and correctly say the abstract gives no rater-agreement figures. The draft infers no benefits or costs beyond what the source supports. 'For an occupation' is a reasonable reading of the occupational-stereotype condition.

> Gendered pronouns followed occupational stereotypes whenever a model committed to one

media — pass: The selected media is the verified link to the original arXiv abstract page. This is the most direct way for readers to reach the source.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/cae90331e0d5bfdbe9ed1a56f362941fa7ca58501a093e54f546ddfa4b501c6d/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-paper-reports-that-language-models-often-answer-cae90331.json)
