# Researchers use models’ confidence to train language models [Canonical reading](https://escapemeridian.com/readings/researchers-use-models-confidence-to-train-language-models-e93717af) Researchers Carel van Niekerk, Renato Vukovic and coauthors present Reinforcement Learning from Self-Feedback (RLSF), a post-training method that uses a language model’s confidence in its own answers to create training preferences. Their abstract starts from a problem: models can give plausible answers that are poorly calibrated, meaning their confidence does not reliably match answer correctness. RLSF generates multiple Chain-of-Thought (CoT) decoding beams from a frozen LLM, scores confidence in each final answer span, and ranks the resulting traces. Those rankings become synthetic preferences for fine-tuning, without human labels, gold answers or externally curated rewards. The authors report improved calibration and step-by-step reasoning, with better results on arithmetic reasoning and multiple-choice question answering. The central idea is to turn a model’s own confidence into a learning signal rather than rely on manually supplied answers or rewards. The abstract describes results on the tasks it names; it does not establish performance across other tasks or real-world reliability. The authors call the method data-efficient, but the abstract does not quantify data efficiency or provide the underlying evaluation details. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Jul 29, 2025 (2025-07-29). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-09T01:55:01.261Z. First observed: 2026-11-06T05:10:59.015Z. Retrieved: 2026-11-09T01:54:17.232Z. - [Post-Training Large Language Models via Reinforcement Learning from Self-Feedback](https://arxiv.org/abs/2507.21931) ## Limits The evidence is the paper’s abstract, which does not provide the evaluation details or quantify the reported gains. The abstract reports results for arithmetic reasoning and multiple-choice question answering; it does not establish performance on other tasks or real-world reliability. ## Open question How large are the reported calibration and task-performance gains, and how were they evaluated? ## Source-stated The authors present RLSF, a post-training method that uses a model's intrinsic confidence as a self-generated reward. > we present Reinforcement Learning from Self-Feedback (RLSF), a post-training stage that utilises the model's intrinsic confidence as a self-generated reward. - [Supporting source](https://arxiv.org/abs/2507.21931) ## Source-stated RLSF creates synthetic preferences by generating multiple reasoning traces from a frozen model, scoring final-answer confidence and ranking the traces. > By generating multiple CoT decoding beams from a frozen LLM, we compute the confidence of each final answer span and rank the resulting traces accordingly to create synthetic preferences. - [Supporting source](https://arxiv.org/abs/2507.21931) ## Source-stated The authors say the method requires no human labels, gold answers or externally curated rewards. > These preferences are subsequently utilised to fine-tune the policy through standard preference optimisation, requiring no human labels, gold answers, or externally curated rewards. - [Supporting source](https://arxiv.org/abs/2507.21931) ## Source-stated The authors report improved calibration and step-by-step reasoning, with better performance on arithmetic reasoning and multiple-choice question answering. > RLSF simultaneously (i) refines the model's probability estimates--restoring well-behaved calibration--and (ii) strengthens step-by-step reasoning, yielding improved performance on arithmetic reasoning and multiple-choice question answering. - [Supporting source](https://arxiv.org/abs/2507.21931) ## Source and contributor context van Niekerk, Carel, Vukovic, Renato, Ruppik, Benjamin, Lin, Hsien-chin, Feng, Shutong, Gašić, Milica · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Reinforcement learning Training by trial and error, in which a system is rewarded for the outcomes its designers want. - [Definition reference](https://developers.google.com/machine-learning/glossary#reinforcement-learning-rl) ## Model calibration Whether a model's stated confidence matches how often it turns out to be right. - [Definition reference](https://en.wikipedia.org/wiki/Calibration_%28statistics%29) ## Chain of thought The step-by-step working a model writes out before giving its final answer. - [Definition reference](https://developers.google.com/machine-learning/glossary#chain-of-thought-prompting) ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Fine-tuning Further training of an existing model to adapt it to a particular task or use. - [Definition reference](https://developers.google.com/machine-learning/glossary#fine-tuning) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: f6ae368f-24c1-480a-80ad-f55b512ceefa. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-09T01:55:01.261Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: cade57d7670ab0df5fb6cf2f17444d9333fa8bdc3c23861332bed85753cea60f. Source text hash: 5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b. ## Revision and source-check history ### Version 1 — 2026-11-09T01:55:01.261Z Researchers Carel van Niekerk, Renato Vukovic and coauthors present Reinforcement Learning from Self-Feedback (RLSF), a post-training method that uses a language model’s confidence in its own answers to create training preferences. Their abstract starts from a problem: models can give plausible answers that are poorly calibrated, meaning their confidence does not reliably match answer correctness. RLSF generates multiple Chain-of-Thought (CoT) decoding beams from a frozen LLM, scores confidence in each final answer span, and ranks the resulting traces. Those rankings become synthetic preferences for fine-tuning, without human labels, gold answers or externally curated rewards. The authors report improved calibration and step-by-step reasoning, with better results on arithmetic reasoning and multiple-choice question answering. The central idea is to turn a model’s own confidence into a learning signal rather than rely on manually supplied answers or rewards. The abstract describes results on the tasks it names; it does not establish performance across other tasks or real-world reliability. The authors call the method data-efficient, but the abstract does not quantify data efficiency or provide the underlying evaluation details. ## Original source arXiv cs.AI · Article Original source date: Jul 29, 2025 (2025-07-29). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-09T01:55:01.261Z. First observed: 2026-11-06T05:10:59.015Z. Retrieved: 2026-11-09T01:54:17.232Z. - [Post-Training Large Language Models via Reinforcement Learning from Self-Feedback](https://arxiv.org/abs/2507.21931) ## Limits The evidence is the paper’s abstract, which does not provide the evaluation details or quantify the reported gains. The abstract reports results for arithmetic reasoning and multiple-choice question answering; it does not establish performance on other tasks or real-world reliability. ## Open question How large are the reported calibration and task-performance gains, and how were they evaluated? ## Source-stated The authors present RLSF, a post-training method that uses a model's intrinsic confidence as a self-generated reward. > we present Reinforcement Learning from Self-Feedback (RLSF), a post-training stage that utilises the model's intrinsic confidence as a self-generated reward. - [Supporting source](https://arxiv.org/abs/2507.21931) ## Source-stated RLSF creates synthetic preferences by generating multiple reasoning traces from a frozen model, scoring final-answer confidence and ranking the traces. > By generating multiple CoT decoding beams from a frozen LLM, we compute the confidence of each final answer span and rank the resulting traces accordingly to create synthetic preferences. - [Supporting source](https://arxiv.org/abs/2507.21931) ## Source-stated The authors say the method requires no human labels, gold answers or externally curated rewards. > These preferences are subsequently utilised to fine-tune the policy through standard preference optimisation, requiring no human labels, gold answers, or externally curated rewards. - [Supporting source](https://arxiv.org/abs/2507.21931) ## Source-stated The authors report improved calibration and step-by-step reasoning, with better performance on arithmetic reasoning and multiple-choice question answering. > RLSF simultaneously (i) refines the model's probability estimates--restoring well-behaved calibration--and (ii) strengthens step-by-step reasoning, yielding improved performance on arithmetic reasoning and multiple-choice question answering. - [Supporting source](https://arxiv.org/abs/2507.21931) ## Source and contributor context van Niekerk, Carel, Vukovic, Renato, Ruppik, Benjamin, Lin, Hsien-chin, Feng, Shutong, Gašić, Milica · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Reinforcement learning Training by trial and error, in which a system is rewarded for the outcomes its designers want. - [Definition reference](https://developers.google.com/machine-learning/glossary#reinforcement-learning-rl) ## Model calibration Whether a model's stated confidence matches how often it turns out to be right. - [Definition reference](https://en.wikipedia.org/wiki/Calibration_%28statistics%29) ## Chain of thought The step-by-step working a model writes out before giving its final answer. - [Definition reference](https://developers.google.com/machine-learning/glossary#chain-of-thought-prompting) ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Fine-tuning Further training of an existing model to adapt it to a particular task or use. - [Definition reference](https://developers.google.com/machine-learning/glossary#fine-tuning) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: f6ae368f-24c1-480a-80ad-f55b512ceefa. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-09T01:55:01.261Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: cade57d7670ab0df5fb6cf2f17444d9333fa8bdc3c23861332bed85753cea60f. Source text hash: 5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b. ### passed — 2026-11-09T01:55:01.261Z — anthropic/claude-opus-5.5 Publication: f6ae368f-24c1-480a-80ad-f55b512ceefa attribution — pass: The reading correctly names Carel van Niekerk and Renato Vukovic among the listed authors and uses 'coauthors' for the rest. Claims are attributed to the authors and their abstract rather than presented as established fact. > Authors:Carel van Niekerk, Renato Vukovic, Benjamin Ruppik, Hsien-chin Lin, Shutong Feng, Milica Gašić summary — pass: The reading leads with the actor and the method, and defines poor calibration in plain language. It states the central idea clearly: the model's own confidence becomes a learning signal instead of human labels. Technical terms such as decoding beams and frozen LLM are kept exact, as the term-library approach intends, and the core idea stays understandable without them. The title and question are accurate and non-leading. The 'data-efficient' claim stays attributed to the authors. > By converting a model's own uncertainty into structured self-feedback, RLSF affirms reinforcement learning on intrinsic model behaviour as a principled and data-efficient component of the LLM post-training pipeline. date — pass: The originalPublished date, 2025-07-29, matches the v1 submission date. It is not taken from the later v3 revision on 5 Nov 2026 or from the feed date. > \[Submitted on 29 Jul 2025 (v1), last revised 5 Nov 2026 (this version, v3)\] support — pass: Every claim excerpt is an exact quote from the abstract and supports its paired claim. The limitations are accurate: the abstract gives no numbers or evaluation details. The reading does not turn the reported task gains into general real-world reliability. It adds no unsupported benefits, costs or stakeholder claims. > yielding improved performance on arithmetic reasoning and multiple-choice question answering media — pass: The selected media is the verified direct link to the arXiv abstract page for this paper, which is the source itself. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/e93717af6f94b9a2b0617f61148d125f00cdb5d0f9cfb70a3bc5fce67efe86fd/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/researchers-use-models-confidence-to-train-language-models-e93717af.json)