{"id":"e93717af6f94b9a2b0617f61148d125f00cdb5d0f9cfb70a3bc5fce67efe86fd","title":"Researchers use models’ confidence to train language models","argument":"Researchers Carel van Niekerk, Renato Vukovic and coauthors present Reinforcement Learning from Self-Feedback (RLSF), a post-training method that uses a language model’s confidence in its own answers to create training preferences. Their abstract starts from a problem: models can give plausible answers that are poorly calibrated, meaning their confidence does not reliably match answer correctness. RLSF generates multiple Chain-of-Thought (CoT) decoding beams from a frozen LLM, scores confidence in each final answer span, and ranks the resulting traces. Those rankings become synthetic preferences for fine-tuning, without human labels, gold answers or externally curated rewards.\n\nThe authors report improved calibration and step-by-step reasoning, with better results on arithmetic reasoning and multiple-choice question answering. The central idea is to turn a model’s own confidence into a learning signal rather than rely on manually supplied answers or rewards. The abstract describes results on the tasks it names; it does not establish performance across other tasks or real-world reliability. The authors call the method data-efficient, but the abstract does not quantify data efficiency or provide the underlying evaluation details.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence is the paper’s abstract, which does not provide the evaluation details or quantify the reported gains. The abstract reports results for arithmetic reasoning and multiple-choice question answering; it does not establish performance on other tasks or real-world reliability.","speaker":{"name":"van Niekerk, Carel, Vukovic, Renato, Ruppik, Benjamin, Lin, Hsien-chin, Feng, Shutong, Gašić, Milica","roleAtSource":"Article author"},"source":{"title":"Post-Training Large Language Models via Reinforcement Learning from Self-Feedback","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2507.21931","publishedAt":"2025-07-29","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors present RLSF, a post-training method that uses a model's intrinsic confidence as a self-generated reward.","excerpt":"we present Reinforcement Learning from Self-Feedback (RLSF), a post-training stage that utilises the model's intrinsic confidence as a self-generated reward.","url":"https://arxiv.org/abs/2507.21931","relationship":"Source-stated"},{"text":"RLSF creates synthetic preferences by generating multiple reasoning traces from a frozen model, scoring final-answer confidence and ranking the traces.","excerpt":"By generating multiple CoT decoding beams from a frozen LLM, we compute the confidence of each final answer span and rank the resulting traces accordingly to create synthetic preferences.","url":"https://arxiv.org/abs/2507.21931","relationship":"Source-stated"},{"text":"The authors say the method requires no human labels, gold answers or externally curated rewards.","excerpt":"These preferences are subsequently utilised to fine-tune the policy through standard preference optimisation, requiring no human labels, gold answers, or externally curated rewards.","url":"https://arxiv.org/abs/2507.21931","relationship":"Source-stated"},{"text":"The authors report improved calibration and step-by-step reasoning, with better performance on arithmetic reasoning and multiple-choice question answering.","excerpt":"RLSF simultaneously (i) refines the model's probability estimates--restoring well-behaved calibration--and (ii) strengthens step-by-step reasoning, yielding improved performance on arithmetic reasoning and multiple-choice question answering.","url":"https://arxiv.org/abs/2507.21931","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2507.21931","checkedAt":"2026-11-09T04:02:08.108Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How large are the reported calibration and task-performance gains, and how were they evaluated?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Researchers use models’ confidence to train language models","summary":"Researchers Carel van Niekerk, Renato Vukovic and coauthors present Reinforcement Learning from Self-Feedback (RLSF), a post-training method that uses a language model’s confidence in its own answers to create training preferences. Their abstract starts from a problem: models can give plausible answers that are poorly calibrated, meaning their confidence does not reliably match answer correctness. RLSF generates multiple Chain-of-Thought (CoT) decoding beams from a frozen LLM, scores confidence in each final answer span, and ranks the resulting traces. Those rankings become synthetic preferences for fine-tuning, without human labels, gold answers or externally curated rewards.\n\nThe authors report improved calibration and step-by-step reasoning, with better results on arithmetic reasoning and multiple-choice question answering. The central idea is to turn a model’s own confidence into a learning signal rather than rely on manually supplied answers or rewards. The abstract describes results on the tasks it names; it does not establish performance across other tasks or real-world reliability. The authors call the method data-efficient, but the abstract does not quantify data efficiency or provide the underlying evaluation details.","question":"How large are the reported calibration and task-performance gains, and how were they evaluated?","claims":[{"text":"The authors present RLSF, a post-training method that uses a model's intrinsic confidence as a self-generated reward.","excerpt":"we present Reinforcement Learning from Self-Feedback (RLSF), a post-training stage that utilises the model's intrinsic confidence as a self-generated reward.","support":"source-stated"},{"text":"RLSF creates synthetic preferences by generating multiple reasoning traces from a frozen model, scoring final-answer confidence and ranking the traces.","excerpt":"By generating multiple CoT decoding beams from a frozen LLM, we compute the confidence of each final answer span and rank the resulting traces accordingly to create synthetic preferences.","support":"source-stated"},{"text":"The authors say the method requires no human labels, gold answers or externally curated rewards.","excerpt":"These preferences are subsequently utilised to fine-tune the policy through standard preference optimisation, requiring no human labels, gold answers, or externally curated rewards.","support":"source-stated"},{"text":"The authors report improved calibration and step-by-step reasoning, with better performance on arithmetic reasoning and multiple-choice question answering.","excerpt":"RLSF simultaneously (i) refines the model's probability estimates--restoring well-behaved calibration--and (ii) strengthens step-by-step reasoning, yielding improved performance on arithmetic reasoning and multiple-choice question answering.","support":"source-stated"}],"limitations":["The evidence is the paper’s abstract, which does not provide the evaluation details or quantify the reported gains.","The abstract reports results for arithmetic reasoning and multiple-choice question answering; it does not establish performance on other tasks or real-world reliability."],"mediaId":"c870d533ea3f83992e9737c52c3d0cddf74d1407223b81498a124707a4826dc0","correctionReason":null,"id":"e93717af6f94b9a2b0617f61148d125f00cdb5d0f9cfb70a3bc5fce67efe86fd","publicationId":"f6ae368f-24c1-480a-80ad-f55b512ceefa","version":1,"sourceUrl":"https://arxiv.org/abs/2507.21931","sourceTitle":"Post-Training Large Language Models via Reinforcement Learning from Self-Feedback","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2025-07-29","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:10:59.015Z","createdAt":"2026-11-09T01:55:01.261Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"c870d533ea3f83992e9737c52c3d0cddf74d1407223b81498a124707a4826dc0","kind":"article","url":"https://arxiv.org/abs/2507.21931","sourceUrl":"https://arxiv.org/abs/2507.21931","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-09T01:54:17.409Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2507.21931","checkedAt":"2026-11-09T04:02:08.108Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"c870d533ea3f83992e9737c52c3d0cddf74d1407223b81498a124707a4826dc0","confidence":0.8300000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"9f50bbcd6a4ff5ee2949d321a45b61330fbd5987d1ca080bda4c8d062b7063e2","decidedAt":"2026-11-09T01:54:18.811Z"},"provenance":{"textHash":"5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b","observationId":"cade57d7670ab0df5fb6cf2f17444d9333fa8bdc3c23861332bed85753cea60f","retrievedAt":"2026-11-09T01:54:17.232Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"12f9ae1f-beb0-4b91-8f8a-67e4bfe2c856"},"reviewer":{"model":"gpt-6-luna","runId":"95653c86-ddcf-4d6b-84b2-c95a1ee46a4d"}},"audit":{"status":"passed","reviewedAt":"2026-11-09T01:55:01.261Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"van Niekerk, Carel","url":null,"kind":"unknown"},{"name":"Vukovic, Renato","url":null,"kind":"unknown"},{"name":"Ruppik, Benjamin","url":null,"kind":"unknown"},{"name":"Lin, Hsien-chin","url":null,"kind":"unknown"},{"name":"Feng, Shutong","url":null,"kind":"unknown"},{"name":"Gašić, Milica","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"van Niekerk, Carel","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"van Niekerk, Carel","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2507.21931","locator":"head/meta[name=author|citation_author]"},"assertionId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T04:02:08.108Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855","sourceUrl":"https://arxiv.org/abs/2507.21931","contentSha256":"52ebb38d18e5b5a461be1f3d1f3c9111503b3d8ecaa3277cf0678224aa908143","textHash":"5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b","observedAt":"2026-11-09T04:02:08.108Z"}},{"actor":{"name":"Vukovic, Renato","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Vukovic, Renato","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2507.21931","locator":"head/meta[name=author|citation_author]"},"assertionId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T04:02:08.108Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855","sourceUrl":"https://arxiv.org/abs/2507.21931","contentSha256":"52ebb38d18e5b5a461be1f3d1f3c9111503b3d8ecaa3277cf0678224aa908143","textHash":"5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b","observedAt":"2026-11-09T04:02:08.108Z"}},{"actor":{"name":"Ruppik, Benjamin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ruppik, Benjamin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2507.21931","locator":"head/meta[name=author|citation_author]"},"assertionId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T04:02:08.108Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855","sourceUrl":"https://arxiv.org/abs/2507.21931","contentSha256":"52ebb38d18e5b5a461be1f3d1f3c9111503b3d8ecaa3277cf0678224aa908143","textHash":"5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b","observedAt":"2026-11-09T04:02:08.108Z"}},{"actor":{"name":"Lin, Hsien-chin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Lin, Hsien-chin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2507.21931","locator":"head/meta[name=author|citation_author]"},"assertionId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T04:02:08.108Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855","sourceUrl":"https://arxiv.org/abs/2507.21931","contentSha256":"52ebb38d18e5b5a461be1f3d1f3c9111503b3d8ecaa3277cf0678224aa908143","textHash":"5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b","observedAt":"2026-11-09T04:02:08.108Z"}},{"actor":{"name":"Feng, Shutong","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Feng, Shutong","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2507.21931","locator":"head/meta[name=author|citation_author]"},"assertionId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T04:02:08.108Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855","sourceUrl":"https://arxiv.org/abs/2507.21931","contentSha256":"52ebb38d18e5b5a461be1f3d1f3c9111503b3d8ecaa3277cf0678224aa908143","textHash":"5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b","observedAt":"2026-11-09T04:02:08.108Z"}},{"actor":{"name":"Gašić, Milica","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gašić, Milica","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2507.21931","locator":"head/meta[name=author|citation_author]"},"assertionId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T04:02:08.108Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855","sourceUrl":"https://arxiv.org/abs/2507.21931","contentSha256":"52ebb38d18e5b5a461be1f3d1f3c9111503b3d8ecaa3277cf0678224aa908143","textHash":"5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b","observedAt":"2026-11-09T04:02:08.108Z"}}],"participationCoverage":{"total":6,"returned":6,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2507.21931","snapshotId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855","contentSha256":"52ebb38d18e5b5a461be1f3d1f3c9111503b3d8ecaa3277cf0678224aa908143","textHash":"5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b","observedAt":"2026-11-09T04:02:08.108Z","basis":"publisher-page-metadata","sourceLinks":48,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/e93717af6f94b9a2b0617f61148d125f00cdb5d0f9cfb70a3bc5fce67efe86fd/source"}},"canonicalUrl":"https://escapemeridian.com/readings/researchers-use-models-confidence-to-train-language-models-e93717af","formats":{"html":"/readings/researchers-use-models-confidence-to-train-language-models-e93717af","markdown":"/readings/researchers-use-models-confidence-to-train-language-models-e93717af.md","json":"/readings/researchers-use-models-confidence-to-train-language-models-e93717af.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-reinforcement-learning","name":"Reinforcement learning","explanation":"Training by trial and error, in which a system is rewarded for the outcomes its designers want.","referenceUrl":"https://developers.google.com/machine-learning/glossary#reinforcement-learning-rl"},{"id":"term-calibration","name":"Model calibration","explanation":"Whether a model's stated confidence matches how often it turns out to be right.","referenceUrl":"https://en.wikipedia.org/wiki/Calibration_(statistics)"},{"id":"term-chain-of-thought","name":"Chain of thought","explanation":"The step-by-step working a model writes out before giving its final answer.","referenceUrl":"https://developers.google.com/machine-learning/glossary#chain-of-thought-prompting"},{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"},{"id":"term-fine-tuning","name":"Fine-tuning","explanation":"Further training of an existing model to adapt it to a particular task or use.","referenceUrl":"https://developers.google.com/machine-learning/glossary#fine-tuning"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"e93717af6f94b9a2b0617f61148d125f00cdb5d0f9cfb70a3bc5fce67efe86fd","revision":"f6ae368f-24c1-480a-80ad-f55b512ceefa","revisionKind":"publication-id","title":"Researchers use models’ confidence to train language models","url":"https://escapemeridian.com/readings/researchers-use-models-confidence-to-train-language-models-e93717af"},"source":{"type":"source","id":"https://arxiv.org/abs/2507.21931","revision":"5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b","revisionKind":"sha256-source-text","title":"Post-Training Large Language Models via Reinforcement Learning from Self-Feedback","url":"https://arxiv.org/abs/2507.21931"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"e93717af6f94b9a2b0617f61148d125f00cdb5d0f9cfb70a3bc5fce67efe86fd","title":"Researchers use models’ confidence to train language models","argument":"Researchers Carel van Niekerk, Renato Vukovic and coauthors present Reinforcement Learning from Self-Feedback (RLSF), a post-training method that uses a language model’s confidence in its own answers to create training preferences. Their abstract starts from a problem: models can give plausible answers that are poorly calibrated, meaning their confidence does not reliably match answer correctness. RLSF generates multiple Chain-of-Thought (CoT) decoding beams from a frozen LLM, scores confidence in each final answer span, and ranks the resulting traces. Those rankings become synthetic preferences for fine-tuning, without human labels, gold answers or externally curated rewards.\n\nThe authors report improved calibration and step-by-step reasoning, with better results on arithmetic reasoning and multiple-choice question answering. The central idea is to turn a model’s own confidence into a learning signal rather than rely on manually supplied answers or rewards. The abstract describes results on the tasks it names; it does not establish performance across other tasks or real-world reliability. The authors call the method data-efficient, but the abstract does not quantify data efficiency or provide the underlying evaluation details.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence is the paper’s abstract, which does not provide the evaluation details or quantify the reported gains. The abstract reports results for arithmetic reasoning and multiple-choice question answering; it does not establish performance on other tasks or real-world reliability.","speaker":{"name":"van Niekerk, Carel, Vukovic, Renato, Ruppik, Benjamin, Lin, Hsien-chin, Feng, Shutong, Gašić, Milica","roleAtSource":"Article author"},"source":{"title":"Post-Training Large Language Models via Reinforcement Learning from Self-Feedback","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2507.21931","publishedAt":"2025-07-29","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors present RLSF, a post-training method that uses a model's intrinsic confidence as a self-generated reward.","excerpt":"we present Reinforcement Learning from Self-Feedback (RLSF), a post-training stage that utilises the model's intrinsic confidence as a self-generated reward.","url":"https://arxiv.org/abs/2507.21931","relationship":"Source-stated"},{"text":"RLSF creates synthetic preferences by generating multiple reasoning traces from a frozen model, scoring final-answer confidence and ranking the traces.","excerpt":"By generating multiple CoT decoding beams from a frozen LLM, we compute the confidence of each final answer span and rank the resulting traces accordingly to create synthetic preferences.","url":"https://arxiv.org/abs/2507.21931","relationship":"Source-stated"},{"text":"The authors say the method requires no human labels, gold answers or externally curated rewards.","excerpt":"These preferences are subsequently utilised to fine-tune the policy through standard preference optimisation, requiring no human labels, gold answers, or externally curated rewards.","url":"https://arxiv.org/abs/2507.21931","relationship":"Source-stated"},{"text":"The authors report improved calibration and step-by-step reasoning, with better performance on arithmetic reasoning and multiple-choice question answering.","excerpt":"RLSF simultaneously (i) refines the model's probability estimates--restoring well-behaved calibration--and (ii) strengthens step-by-step reasoning, yielding improved performance on arithmetic reasoning and multiple-choice question answering.","url":"https://arxiv.org/abs/2507.21931","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2507.21931","checkedAt":"2026-11-09T04:02:08.108Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How large are the reported calibration and task-performance gains, and how were they evaluated?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Researchers use models’ confidence to train language models","summary":"Researchers Carel van Niekerk, Renato Vukovic and coauthors present Reinforcement Learning from Self-Feedback (RLSF), a post-training method that uses a language model’s confidence in its own answers to create training preferences. Their abstract starts from a problem: models can give plausible answers that are poorly calibrated, meaning their confidence does not reliably match answer correctness. RLSF generates multiple Chain-of-Thought (CoT) decoding beams from a frozen LLM, scores confidence in each final answer span, and ranks the resulting traces. Those rankings become synthetic preferences for fine-tuning, without human labels, gold answers or externally curated rewards.\n\nThe authors report improved calibration and step-by-step reasoning, with better results on arithmetic reasoning and multiple-choice question answering. The central idea is to turn a model’s own confidence into a learning signal rather than rely on manually supplied answers or rewards. The abstract describes results on the tasks it names; it does not establish performance across other tasks or real-world reliability. The authors call the method data-efficient, but the abstract does not quantify data efficiency or provide the underlying evaluation details.","question":"How large are the reported calibration and task-performance gains, and how were they evaluated?","claims":[{"text":"The authors present RLSF, a post-training method that uses a model's intrinsic confidence as a self-generated reward.","excerpt":"we present Reinforcement Learning from Self-Feedback (RLSF), a post-training stage that utilises the model's intrinsic confidence as a self-generated reward.","support":"source-stated"},{"text":"RLSF creates synthetic preferences by generating multiple reasoning traces from a frozen model, scoring final-answer confidence and ranking the traces.","excerpt":"By generating multiple CoT decoding beams from a frozen LLM, we compute the confidence of each final answer span and rank the resulting traces accordingly to create synthetic preferences.","support":"source-stated"},{"text":"The authors say the method requires no human labels, gold answers or externally curated rewards.","excerpt":"These preferences are subsequently utilised to fine-tune the policy through standard preference optimisation, requiring no human labels, gold answers, or externally curated rewards.","support":"source-stated"},{"text":"The authors report improved calibration and step-by-step reasoning, with better performance on arithmetic reasoning and multiple-choice question answering.","excerpt":"RLSF simultaneously (i) refines the model's probability estimates--restoring well-behaved calibration--and (ii) strengthens step-by-step reasoning, yielding improved performance on arithmetic reasoning and multiple-choice question answering.","support":"source-stated"}],"limitations":["The evidence is the paper’s abstract, which does not provide the evaluation details or quantify the reported gains.","The abstract reports results for arithmetic reasoning and multiple-choice question answering; it does not establish performance on other tasks or real-world reliability."],"mediaId":"c870d533ea3f83992e9737c52c3d0cddf74d1407223b81498a124707a4826dc0","correctionReason":null,"id":"e93717af6f94b9a2b0617f61148d125f00cdb5d0f9cfb70a3bc5fce67efe86fd","publicationId":"f6ae368f-24c1-480a-80ad-f55b512ceefa","version":1,"sourceUrl":"https://arxiv.org/abs/2507.21931","sourceTitle":"Post-Training Large Language Models via Reinforcement Learning from Self-Feedback","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2025-07-29","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:10:59.015Z","createdAt":"2026-11-09T01:55:01.261Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"c870d533ea3f83992e9737c52c3d0cddf74d1407223b81498a124707a4826dc0","kind":"article","url":"https://arxiv.org/abs/2507.21931","sourceUrl":"https://arxiv.org/abs/2507.21931","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-09T01:54:17.409Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2507.21931","checkedAt":"2026-11-09T04:02:08.108Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"c870d533ea3f83992e9737c52c3d0cddf74d1407223b81498a124707a4826dc0","confidence":0.8300000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"9f50bbcd6a4ff5ee2949d321a45b61330fbd5987d1ca080bda4c8d062b7063e2","decidedAt":"2026-11-09T01:54:18.811Z"},"provenance":{"textHash":"5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b","observationId":"cade57d7670ab0df5fb6cf2f17444d9333fa8bdc3c23861332bed85753cea60f","retrievedAt":"2026-11-09T01:54:17.232Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"12f9ae1f-beb0-4b91-8f8a-67e4bfe2c856"},"reviewer":{"model":"gpt-6-luna","runId":"95653c86-ddcf-4d6b-84b2-c95a1ee46a4d"}},"audit":{"status":"passed","reviewedAt":"2026-11-09T01:55:01.261Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"van Niekerk, Carel","url":null,"kind":"unknown"},{"name":"Vukovic, Renato","url":null,"kind":"unknown"},{"name":"Ruppik, Benjamin","url":null,"kind":"unknown"},{"name":"Lin, Hsien-chin","url":null,"kind":"unknown"},{"name":"Feng, Shutong","url":null,"kind":"unknown"},{"name":"Gašić, Milica","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"van Niekerk, Carel","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"van Niekerk, Carel","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2507.21931","locator":"head/meta[name=author|citation_author]"},"assertionId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T04:02:08.108Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855","sourceUrl":"https://arxiv.org/abs/2507.21931","contentSha256":"52ebb38d18e5b5a461be1f3d1f3c9111503b3d8ecaa3277cf0678224aa908143","textHash":"5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b","observedAt":"2026-11-09T04:02:08.108Z"}},{"actor":{"name":"Vukovic, Renato","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Vukovic, Renato","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2507.21931","locator":"head/meta[name=author|citation_author]"},"assertionId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T04:02:08.108Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855","sourceUrl":"https://arxiv.org/abs/2507.21931","contentSha256":"52ebb38d18e5b5a461be1f3d1f3c9111503b3d8ecaa3277cf0678224aa908143","textHash":"5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b","observedAt":"2026-11-09T04:02:08.108Z"}},{"actor":{"name":"Ruppik, Benjamin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ruppik, Benjamin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2507.21931","locator":"head/meta[name=author|citation_author]"},"assertionId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T04:02:08.108Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855","sourceUrl":"https://arxiv.org/abs/2507.21931","contentSha256":"52ebb38d18e5b5a461be1f3d1f3c9111503b3d8ecaa3277cf0678224aa908143","textHash":"5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b","observedAt":"2026-11-09T04:02:08.108Z"}},{"actor":{"name":"Lin, Hsien-chin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Lin, Hsien-chin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2507.21931","locator":"head/meta[name=author|citation_author]"},"assertionId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T04:02:08.108Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855","sourceUrl":"https://arxiv.org/abs/2507.21931","contentSha256":"52ebb38d18e5b5a461be1f3d1f3c9111503b3d8ecaa3277cf0678224aa908143","textHash":"5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b","observedAt":"2026-11-09T04:02:08.108Z"}},{"actor":{"name":"Feng, Shutong","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Feng, Shutong","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2507.21931","locator":"head/meta[name=author|citation_author]"},"assertionId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T04:02:08.108Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855","sourceUrl":"https://arxiv.org/abs/2507.21931","contentSha256":"52ebb38d18e5b5a461be1f3d1f3c9111503b3d8ecaa3277cf0678224aa908143","textHash":"5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b","observedAt":"2026-11-09T04:02:08.108Z"}},{"actor":{"name":"Gašić, Milica","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gašić, Milica","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2507.21931","locator":"head/meta[name=author|citation_author]"},"assertionId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T04:02:08.108Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855","sourceUrl":"https://arxiv.org/abs/2507.21931","contentSha256":"52ebb38d18e5b5a461be1f3d1f3c9111503b3d8ecaa3277cf0678224aa908143","textHash":"5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b","observedAt":"2026-11-09T04:02:08.108Z"}}],"participationCoverage":{"total":6,"returned":6,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2507.21931","snapshotId":"96212f86482a1dd8b92f2594bc5811f80ee4e12442ff9a1a39a5494694627855","contentSha256":"52ebb38d18e5b5a461be1f3d1f3c9111503b3d8ecaa3277cf0678224aa908143","textHash":"5b14eb4edf48eec5f84fa66a806b5d8deb3999d93bbd0cde6146332b4134986b","observedAt":"2026-11-09T04:02:08.108Z","basis":"publisher-page-metadata","sourceLinks":48,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/e93717af6f94b9a2b0617f61148d125f00cdb5d0f9cfb70a3bc5fce67efe86fd/source"}}}],"audits":[{"publicationId":"f6ae368f-24c1-480a-80ad-f55b512ceefa","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading correctly names Carel van Niekerk and Renato Vukovic among the listed authors and uses 'coauthors' for the rest. Claims are attributed to the authors and their abstract rather than presented as established fact.","excerpt":"Authors:Carel van Niekerk, Renato Vukovic, Benjamin Ruppik, Hsien-chin Lin, Shutong Feng, Milica Gašić"},{"dimension":"summary","status":"pass","explanation":"The reading leads with the actor and the method, and defines poor calibration in plain language. It states the central idea clearly: the model's own confidence becomes a learning signal instead of human labels. Technical terms such as decoding beams and frozen LLM are kept exact, as the term-library approach intends, and the core idea stays understandable without them. The title and question are accurate and non-leading. The 'data-efficient' claim stays attributed to the authors.","excerpt":"By converting a model's own uncertainty into structured self-feedback, RLSF affirms reinforcement learning on intrinsic model behaviour as a principled and data-efficient component of the LLM post-training pipeline."},{"dimension":"date","status":"pass","explanation":"The originalPublished date, 2025-07-29, matches the v1 submission date. It is not taken from the later v3 revision on 5 Nov 2026 or from the feed date.","excerpt":"[Submitted on 29 Jul 2025 (v1), last revised 5 Nov 2026 (this version, v3)]"},{"dimension":"support","status":"pass","explanation":"Every claim excerpt is an exact quote from the abstract and supports its paired claim. The limitations are accurate: the abstract gives no numbers or evaluation details. The reading does not turn the reported task gains into general real-world reliability. It adds no unsupported benefits, costs or stakeholder claims.","excerpt":"yielding improved performance on arithmetic reasoning and multiple-choice question answering"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the arXiv abstract page for this paper, which is the source itself.","excerpt":null}],"createdAt":"2026-11-09T01:55:01.261Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}