van Niekerk, Carel, Vukovic, Renato, Ruppik, Benjamin, Lin, Hsien-chin, Feng, Shutong, Gašić, Milica · Jul 29, 2025
Researchers use models’ confidence to train language models
Markdown · JSON · Original source
Researchers Carel van Niekerk, Renato Vukovic and coauthors present from Self-Feedback (RLSF), a post-training method that uses a language model’s confidence in its own answers to create training preferences. Their abstract starts from a problem: models can give plausible answers that are , meaning their confidence does not reliably match answer correctness. RLSF generates multiple (CoT) decoding beams from a frozen , scores confidence in each final answer span, and ranks the resulting traces. Those rankings become synthetic preferences for , without human labels, gold answers or externally curated rewards. The authors report improved calibration and step-by-step reasoning, with better results on arithmetic reasoning and multiple-choice question answering. The central idea is to turn a model’s own confidence into a learning signal rather than rely on manually supplied answers or rewards. The abstract describes results on the tasks it names; it does not establish performance across other tasks or real-world reliability. The authors call the method data-efficient, but the abstract does not quantify data efficiency or provide the underlying evaluation details.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Jul 29, 2025 (2025-07-29).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T01:55:01.261Z. First observed: 2026-11-06T05:10:59.015Z. Retrieved: 2026-11-09T01:54:17.232Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The evidence is the paper’s abstract, which does not provide the evaluation details or quantify the reported gains. The abstract reports results for arithmetic reasoning and multiple-choice question answering; it does not establish performance on other tasks or real-world reliability.
Open question
How large are the reported calibration and task-performance gains, and how were they evaluated?
Source-stated
The authors present RLSF, a post-training method that uses a model's intrinsic confidence as a self-generated reward.
we present Reinforcement Learning from Self-Feedback (RLSF), a post-training stage that utilises the model's intrinsic confidence as a self-generated reward.
Source-stated
RLSF creates synthetic preferences by generating multiple reasoning traces from a frozen model, scoring final-answer confidence and ranking the traces.
By generating multiple CoT decoding beams from a frozen LLM, we compute the confidence of each final answer span and rank the resulting traces accordingly to create synthetic preferences.
Source-stated
The authors say the method requires no human labels, gold answers or externally curated rewards.
These preferences are subsequently utilised to fine-tune the policy through standard preference optimisation, requiring no human labels, gold answers, or externally curated rewards.
Source-stated
The authors report improved calibration and step-by-step reasoning, with better performance on arithmetic reasoning and multiple-choice question answering.
RLSF simultaneously (i) refines the model's probability estimates--restoring well-behaved calibration--and (ii) strengthens step-by-step reasoning, yielding improved performance on arithmetic reasoning and multiple-choice question answering.
Source and contributor context
van Niekerk, Carel, Vukovic, Renato, Ruppik, Benjamin, Lin, Hsien-chin, Feng, Shutong, Gašić, Milica · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Reinforcement learning
Training by trial and error, in which a system is rewarded for the outcomes its designers want.
Model calibration
Whether a model's stated confidence matches how often it turns out to be right.
Chain of thought
The step-by-step working a model writes out before giving its final answer.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Fine-tuning
Further training of an existing model to adapt it to a particular task or use.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: f6ae368f-24c1-480a-80ad-f55b512ceefa.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-09T01:55:01.261Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: cade57d7670ab0df5fb6cf2f17444d9333fa8bdc3c23861332bed85753cea60f. Source text hash: 5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b.
Revision and source-check history
Version 1 · 2026-11-09T01:55:01.261Z
Researchers use models’ confidence to train language models
Researchers Carel van Niekerk, Renato Vukovic and coauthors present from Self-Feedback (RLSF), a post-training method that uses a language model’s confidence in its own answers to create training preferences. Their abstract starts from a problem: models can give plausible answers that are , meaning their confidence does not reliably match answer correctness. RLSF generates multiple (CoT) decoding beams from a frozen , scores confidence in each final answer span, and ranks the resulting traces. Those rankings become synthetic preferences for , without human labels, gold answers or externally curated rewards. The authors report improved calibration and step-by-step reasoning, with better results on arithmetic reasoning and multiple-choice question answering. The central idea is to turn a model’s own confidence into a learning signal rather than rely on manually supplied answers or rewards. The abstract describes results on the tasks it names; it does not establish performance across other tasks or real-world reliability. The authors call the method data-efficient, but the abstract does not quantify data efficiency or provide the underlying evaluation details.
Original source
cs.AI · Article
Source details
Original source date: Jul 29, 2025 (2025-07-29).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T01:55:01.261Z. First observed: 2026-11-06T05:10:59.015Z. Retrieved: 2026-11-09T01:54:17.232Z.
Limits
The evidence is the paper’s abstract, which does not provide the evaluation details or quantify the reported gains. The abstract reports results for arithmetic reasoning and multiple-choice question answering; it does not establish performance on other tasks or real-world reliability.
Open question
How large are the reported calibration and task-performance gains, and how were they evaluated?
Source-stated
The authors present RLSF, a post-training method that uses a model's intrinsic confidence as a self-generated reward.
we present Reinforcement Learning from Self-Feedback (RLSF), a post-training stage that utilises the model's intrinsic confidence as a self-generated reward.
Source-stated
RLSF creates synthetic preferences by generating multiple reasoning traces from a frozen model, scoring final-answer confidence and ranking the traces.
By generating multiple CoT decoding beams from a frozen LLM, we compute the confidence of each final answer span and rank the resulting traces accordingly to create synthetic preferences.
Source-stated
The authors say the method requires no human labels, gold answers or externally curated rewards.
These preferences are subsequently utilised to fine-tune the policy through standard preference optimisation, requiring no human labels, gold answers, or externally curated rewards.
Source-stated
The authors report improved calibration and step-by-step reasoning, with better performance on arithmetic reasoning and multiple-choice question answering.
RLSF simultaneously (i) refines the model's probability estimates--restoring well-behaved calibration--and (ii) strengthens step-by-step reasoning, yielding improved performance on arithmetic reasoning and multiple-choice question answering.
Source and contributor context
van Niekerk, Carel, Vukovic, Renato, Ruppik, Benjamin, Lin, Hsien-chin, Feng, Shutong, Gašić, Milica · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Reinforcement learning
Training by trial and error, in which a system is rewarded for the outcomes its designers want.
Model calibration
Whether a model's stated confidence matches how often it turns out to be right.
Chain of thought
The step-by-step working a model writes out before giving its final answer.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Fine-tuning
Further training of an existing model to adapt it to a particular task or use.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: f6ae368f-24c1-480a-80ad-f55b512ceefa.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-09T01:55:01.261Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: cade57d7670ab0df5fb6cf2f17444d9333fa8bdc3c23861332bed85753cea60f. Source text hash: 5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b.
passed · 2026-11-09T01:55:01.261Z · anthropic/claude-opus-5.5
Publication: f6ae368f-24c1-480a-80ad-f55b512ceefa
attribution — pass: The reading correctly names Carel van Niekerk and Renato Vukovic among the listed authors and uses 'coauthors' for the rest. Claims are attributed to the authors and their abstract rather than presented as established fact.
Authors:Carel van Niekerk, Renato Vukovic, Benjamin Ruppik, Hsien-chin Lin, Shutong Feng, Milica Gašić
summary — pass: The reading leads with the actor and the method, and defines poor calibration in plain language. It states the central idea clearly: the model's own confidence becomes a learning signal instead of human labels. Technical terms such as decoding beams and frozen LLM are kept exact, as the term-library approach intends, and the core idea stays understandable without them. The title and question are accurate and non-leading. The 'data-efficient' claim stays attributed to the authors.
By converting a model's own uncertainty into structured self-feedback, RLSF affirms reinforcement learning on intrinsic model behaviour as a principled and data-efficient component of the LLM post-training pipeline.
date — pass: The originalPublished date, 2025-07-29, matches the v1 submission date. It is not taken from the later v3 revision on 5 Nov 2026 or from the feed date.
[Submitted on 29 Jul 2025 (v1), last revised 5 Nov 2026 (this version, v3)]
support — pass: Every claim excerpt is an exact quote from the abstract and supports its paired claim. The limitations are accurate: the abstract gives no numbers or evaluation details. The reading does not turn the reported task gains into general real-world reliability. It adds no unsupported benefits, costs or stakeholder claims.
yielding improved performance on arithmetic reasoning and multiple-choice question answering
media — pass: The selected media is the verified direct link to the arXiv abstract page for this paper, which is the source itself.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Training by trial and error, in which a system is rewarded for the outcomes its designers want.
Definition referenceWhether a model's stated confidence matches how often it turns out to be right.
Definition referenceThe step-by-step working a model writes out before giving its final answer.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceFurther training of an existing model to adapt it to a particular task or use.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivTraining by trial and error, in which a system is rewarded for the outcomes its designers want.
Definition referenceWhether a model's stated confidence matches how often it turns out to be right.
Definition referenceThe step-by-step working a model writes out before giving its final answer.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceFurther training of an existing model to adapt it to a particular task or use.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv