Schwager, Nils, Münker, Simon, Plum, Alistair, Rettinger, Achim · Nov 5, 2026
arXiv cs.AI: Researchers test token probabilities as confidence estimates
Markdown · JSON · Original source
Authors Nils Schwager, Simon Münker, Alistair Plum and Achim Rettinger argue that accuracy alone gives an incomplete picture of language models simulating human behavior. For noisy digital trace data, they say, model performance also depends on calibration: whether stated confidence corresponds to how often predictions are correct. Their case study tests whether token probabilities can estimate confidence when predicting how users of a Luxembourgish news platform rate comments, based on prior interactions. The authors report that the tested models, using in-context learning only, were overconfident and were outperformed by a simple statistical baseline on both accuracy and calibration. Their results also show that accuracy and calibration can move in opposite directions: scaling improved accuracy but degraded calibration, while longer behavioral histories raised accuracy as confidence declined. Task-specific supervised reduced Expected Calibration Error and improved accuracy in this case study. The authors conclude that confidence estimation should be evaluated alongside accuracy when assessing -based agents; a correct prediction alone does not show whether a model’s confidence is .
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 5, 2026 (2026-11-05).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T01:33:37.646Z. First observed: 2026-11-06T05:10:23.941Z. Retrieved: 2026-11-09T01:32:51.297Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The evidence provided is the abstract, which describes a case study on Luxembourgish Next Like Prediction; it does not provide the paper’s methods or detailed results. The abstract does not establish whether the reported findings apply to other human-behavior simulation tasks.
Open question
Do the authors’ findings about token probabilities as confidence estimates hold on human-behavior simulation tasks beyond Luxembourgish Next Like Prediction?
Source-stated
The authors argue that accuracy alone is insufficient to assess LLM simulations of noisy human behavior; calibration should also be assessed.
We argue that on noisy digital trace data, a model's performance depends not only on predicting the correct outcome but on its calibration.
Source-stated
In the Luxembourgish Next Like Prediction case study, the tested models using in-context learning only were overconfident and inferior to a simple statistical baseline on accuracy and calibration.
with in-context learning only, the tested models are overconfident and are outperformed on accuracy and calibration by a simple statistical baseline.
Source-stated
The reported relationship between accuracy and calibration varied across model scaling and behavioral-history length.
scaling from 3B to 8B improves accuracy while degrading calibration, and longer behavioral histories raise accuracy while confidence declines.
Source-stated
Task-specific supervised fine-tuning reduced Expected Calibration Error and improved accuracy in the case study.
reducing Expected Calibration Error from 0.2 to 0.006 while improving accuracy from 0.536 to 0.758.
Source-stated
The authors conclude that confidence estimation should be evaluated alongside accuracy for LLM-based agents.
confidence estimation should be evaluated alongside accuracy to improve validity when using LLM-based agents.
Source and contributor context
Schwager, Nils, Münker, Simon, Plum, Alistair, Rettinger, Achim · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Fine-tuning
Further training of an existing model to adapt it to a particular task or use.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Model calibration
Whether a model's stated confidence matches how often it turns out to be right.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: c696c7af-bd0e-4a58-8cb2-ea6fa633f20c.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-09T01:33:37.646Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: useful_perspective.
Observation: a634eef45fe47b871c54bfa2ae19403d13982e64251d8a09b898cc8117d560ce. Source text hash: 8de4230ee9ad59ece813eac21b69c3d0639b0b8b21fcbfe8983992f77c934d06.
Revision and source-check history
Version 1 · 2026-11-09T01:33:37.646Z
arXiv cs.AI: Researchers test token probabilities as confidence estimates
Authors Nils Schwager, Simon Münker, Alistair Plum and Achim Rettinger argue that accuracy alone gives an incomplete picture of language models simulating human behavior. For noisy digital trace data, they say, model performance also depends on calibration: whether stated confidence corresponds to how often predictions are correct. Their case study tests whether token probabilities can estimate confidence when predicting how users of a Luxembourgish news platform rate comments, based on prior interactions. The authors report that the tested models, using in-context learning only, were overconfident and were outperformed by a simple statistical baseline on both accuracy and calibration. Their results also show that accuracy and calibration can move in opposite directions: scaling improved accuracy but degraded calibration, while longer behavioral histories raised accuracy as confidence declined. Task-specific supervised reduced Expected Calibration Error and improved accuracy in this case study. The authors conclude that confidence estimation should be evaluated alongside accuracy when assessing -based agents; a correct prediction alone does not show whether a model’s confidence is .
Original source
cs.AI · Article
Source details
Original source date: Nov 5, 2026 (2026-11-05).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T01:33:37.646Z. First observed: 2026-11-06T05:10:23.941Z. Retrieved: 2026-11-09T01:32:51.297Z.
Limits
The evidence provided is the abstract, which describes a case study on Luxembourgish Next Like Prediction; it does not provide the paper’s methods or detailed results. The abstract does not establish whether the reported findings apply to other human-behavior simulation tasks.
Open question
Do the authors’ findings about token probabilities as confidence estimates hold on human-behavior simulation tasks beyond Luxembourgish Next Like Prediction?
Source-stated
The authors argue that accuracy alone is insufficient to assess LLM simulations of noisy human behavior; calibration should also be assessed.
We argue that on noisy digital trace data, a model's performance depends not only on predicting the correct outcome but on its calibration.
Source-stated
In the Luxembourgish Next Like Prediction case study, the tested models using in-context learning only were overconfident and inferior to a simple statistical baseline on accuracy and calibration.
with in-context learning only, the tested models are overconfident and are outperformed on accuracy and calibration by a simple statistical baseline.
Source-stated
The reported relationship between accuracy and calibration varied across model scaling and behavioral-history length.
scaling from 3B to 8B improves accuracy while degrading calibration, and longer behavioral histories raise accuracy while confidence declines.
Source-stated
Task-specific supervised fine-tuning reduced Expected Calibration Error and improved accuracy in the case study.
reducing Expected Calibration Error from 0.2 to 0.006 while improving accuracy from 0.536 to 0.758.
Source-stated
The authors conclude that confidence estimation should be evaluated alongside accuracy for LLM-based agents.
confidence estimation should be evaluated alongside accuracy to improve validity when using LLM-based agents.
Source and contributor context
Schwager, Nils, Münker, Simon, Plum, Alistair, Rettinger, Achim · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Fine-tuning
Further training of an existing model to adapt it to a particular task or use.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Model calibration
Whether a model's stated confidence matches how often it turns out to be right.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: c696c7af-bd0e-4a58-8cb2-ea6fa633f20c.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-09T01:33:37.646Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: useful_perspective.
Observation: a634eef45fe47b871c54bfa2ae19403d13982e64251d8a09b898cc8117d560ce. Source text hash: 8de4230ee9ad59ece813eac21b69c3d0639b0b8b21fcbfe8983992f77c934d06.
passed · 2026-11-09T01:33:37.646Z · anthropic/claude-opus-5.5
Publication: c696c7af-bd0e-4a58-8cb2-ea6fa633f20c
attribution — pass: The reading names the four listed authors correctly. It keeps the argument and results attributed to them ('argue', 'report', 'conclude') and identifies arXiv as the publisher.
Authors:Nils Schwager, Simon Münker, Alistair Plum, Achim Rettinger
summary — pass: The summary faithfully conveys the thesis, the task, the main findings and the conclusion, and it explains calibration in plain language. Two optional repairs would help general readers. First, name the tested models (Ministral 3, 3B and 8B), because 'scaling' alone is vague. Second, add tooltips for token probabilities, in-context learning, supervised fine-tuning and Expected Calibration Error. The question follows from the stated scope limitation.
Evaluating Ministral 3 (3B, 8B), we find that, with in-context learning only, the tested models are overconfident
date — pass: The original publication date of 2026-11-05 matches the arXiv v1 submission date. The later feed timestamp was correctly not used.
[Submitted on 5 Nov 2026]
support — pass: Each claim is source-stated, and each excerpt is exact. The opposite movement of accuracy and calibration, the fine-tuning gains and the conclusion are all stated in the abstract. The limitations correctly note that the evidence is abstract-only and that generalization beyond this task is not established. The reading adds no invented benefits or costs.
Task-specific supervised fine-tuning acts as an effective calibration intervention, reducing Expected Calibration Error from 0.2 to 0.006 while improving accuracy from 0.536 to 0.758.
media — pass: The selected media is the verified arXiv abstract page for this paper. It is the direct original source and the most relevant link.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Further training of an existing model to adapt it to a particular task or use.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceWhether a model's stated confidence matches how often it turns out to be right.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivFurther training of an existing model to adapt it to a particular task or use.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceWhether a model's stated confidence matches how often it turns out to be right.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv