# arXiv: Ordinal Credit Assignment: Learning from comparisons, with a judge’s limits [Canonical reading](https://escapemeridian.com/readings/arxiv-ordinal-credit-assignment-learning-from-comparisons-with-72f879f2) Hao Yu’s paper studies multi-step reinforcement learning using comparisons between outcomes rather than a reward function or reward model. It proposes crediting each action by how its reward-to-go compares with other trajectories still running at the same time step. In deterministic environments, the expected update ranks actions by true value; with stochastic outcomes, it favors actions that win more often, not necessarily those with higher mean. In a pre-registered study, comparisons cost no detectable performance relative to scalar returns at the same granularity, and learning a reward model from those comparisons did not do better. With real LLM judges, comparisons were 7–10 points more accurate than pointwise scores. But when training with an LLM judge on a requirement omitted by a hand-written reward, policies satisfied the requirement in all five seeds while landing far less often than with oracle comparisons. The paper attributes that gap to the judge being near chance on near-tied failures that dominate early training. The abstract also reports that comparisons do not close the gap to PPO with scalar rewards on HalfCheetah. These results make the judge’s ability to distinguish close outcomes central to whether comparison-based training works in the reported setting. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: May 7, 2026 (2026-05-07). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-09T12:34:10.445Z. First observed: 2026-11-05T05:00:23.346Z. Retrieved: 2026-11-09T12:33:01.777Z. - [Ordinal Credit Assignment: Multi-Step Reinforcement Learning from Comparisons Alone](https://arxiv.org/abs/2605.06375) ## Limits The supplied evidence is the abstract, so it does not provide the full experimental setup or detail behind the reported results. The reported study covers six environments with ten seeds per claim; the LLM-judge training result covers five seeds. The abstract reports that the judge is near chance on near-tied failures, but does not establish whether improving that accuracy would reduce the performance gap. ## Open question Would a judge that is more accurate on near-tied failures reduce the gap between LLM-judge and oracle-comparison training? ## Source-stated The paper studies multi-step reinforcement learning from comparisons, without a reward function or reward model. > We study reinforcement learning that receives only such comparisons - no reward function, no reward model - in multi-step tasks. - [Supporting source](https://arxiv.org/abs/2605.06375) ## Source-stated Its proposed update credits actions by comparing reward-to-go against other trajectories still running at that time step. > each action is credited by how its reward-to-go compares with those of other trajectories at the same time step - [Supporting source](https://arxiv.org/abs/2605.06375) ## Source-stated The paper says the update ranks actions by true value in deterministic environments, but in stochastic outcomes favors actions that win more often over those with higher mean. > under stochastic outcomes it prefers actions that win more often rather than those with higher mean - [Supporting source](https://arxiv.org/abs/2605.06375) ## Source-stated In the reported study, comparisons had no detectable performance cost relative to scalar returns at the same granularity, and learning a reward model from those comparisons was not better. > comparisons cost no detectable performance relative to scalar returns at the same granularity, learning a reward model from the same comparisons is not better - [Supporting source](https://arxiv.org/abs/2605.06375) ## Source-stated With real LLM judges, comparisons were 7–10 points more accurate than pointwise scores. > With real LLM judges, comparisons are 7-10 points more accurate than pointwise scores - [Supporting source](https://arxiv.org/abs/2605.06375) ## Source-stated In training on a requirement omitted by the hand-written reward, policies satisfied it in all five seeds but landed far less often than with oracle comparisons; the paper attributes the gap to the judge’s poor performance on near-tied failures. > policies satisfy it in all five seeds but land far less often than with oracle comparisons, because the judge is near chance on the near-tied failures dominating early training - [Supporting source](https://arxiv.org/abs/2605.06375) ## Source-stated The paper says comparisons do not close the gap to PPO with scalar rewards on HalfCheetah. > Comparisons do not close the gap to PPO with scalar rewards on HalfCheetah. - [Supporting source](https://arxiv.org/abs/2605.06375) ## Source and contributor context Yu, Hao · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Reinforcement learning Training by trial and error, in which a system is rewarded for the outcomes its designers want. - [Definition reference](https://developers.google.com/machine-learning/glossary#reinforcement-learning-rl) ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## TIME TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services. - [Explore this context](/entities/time) - [Reference](https://time.com/) ## Editorial provenance Version 1. Publication ID: 9d6c1dd0-b1ee-4672-8135-06bd48c4ef53. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T12:34:10.445Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 07910f652aded99fd0146609d228998832041b852d7c89e6496d890bd7911bf0. Source text hash: 840c9e01a25a716fe5c21509f6625be7b44077aece4ac6d187e31cad24106ea1. ## Revision and source-check history ### Version 1 — 2026-11-09T12:34:10.445Z Hao Yu’s paper studies multi-step reinforcement learning using comparisons between outcomes rather than a reward function or reward model. It proposes crediting each action by how its reward-to-go compares with other trajectories still running at the same time step. In deterministic environments, the expected update ranks actions by true value; with stochastic outcomes, it favors actions that win more often, not necessarily those with higher mean. In a pre-registered study, comparisons cost no detectable performance relative to scalar returns at the same granularity, and learning a reward model from those comparisons did not do better. With real LLM judges, comparisons were 7–10 points more accurate than pointwise scores. But when training with an LLM judge on a requirement omitted by a hand-written reward, policies satisfied the requirement in all five seeds while landing far less often than with oracle comparisons. The paper attributes that gap to the judge being near chance on near-tied failures that dominate early training. The abstract also reports that comparisons do not close the gap to PPO with scalar rewards on HalfCheetah. These results make the judge’s ability to distinguish close outcomes central to whether comparison-based training works in the reported setting. ## Original source arXiv cs.AI · Article Original source date: May 7, 2026 (2026-05-07). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-09T12:34:10.445Z. First observed: 2026-11-05T05:00:23.346Z. Retrieved: 2026-11-09T12:33:01.777Z. - [Ordinal Credit Assignment: Multi-Step Reinforcement Learning from Comparisons Alone](https://arxiv.org/abs/2605.06375) ## Limits The supplied evidence is the abstract, so it does not provide the full experimental setup or detail behind the reported results. The reported study covers six environments with ten seeds per claim; the LLM-judge training result covers five seeds. The abstract reports that the judge is near chance on near-tied failures, but does not establish whether improving that accuracy would reduce the performance gap. ## Open question Would a judge that is more accurate on near-tied failures reduce the gap between LLM-judge and oracle-comparison training? ## Source-stated The paper studies multi-step reinforcement learning from comparisons, without a reward function or reward model. > We study reinforcement learning that receives only such comparisons - no reward function, no reward model - in multi-step tasks. - [Supporting source](https://arxiv.org/abs/2605.06375) ## Source-stated Its proposed update credits actions by comparing reward-to-go against other trajectories still running at that time step. > each action is credited by how its reward-to-go compares with those of other trajectories at the same time step - [Supporting source](https://arxiv.org/abs/2605.06375) ## Source-stated The paper says the update ranks actions by true value in deterministic environments, but in stochastic outcomes favors actions that win more often over those with higher mean. > under stochastic outcomes it prefers actions that win more often rather than those with higher mean - [Supporting source](https://arxiv.org/abs/2605.06375) ## Source-stated In the reported study, comparisons had no detectable performance cost relative to scalar returns at the same granularity, and learning a reward model from those comparisons was not better. > comparisons cost no detectable performance relative to scalar returns at the same granularity, learning a reward model from the same comparisons is not better - [Supporting source](https://arxiv.org/abs/2605.06375) ## Source-stated With real LLM judges, comparisons were 7–10 points more accurate than pointwise scores. > With real LLM judges, comparisons are 7-10 points more accurate than pointwise scores - [Supporting source](https://arxiv.org/abs/2605.06375) ## Source-stated In training on a requirement omitted by the hand-written reward, policies satisfied it in all five seeds but landed far less often than with oracle comparisons; the paper attributes the gap to the judge’s poor performance on near-tied failures. > policies satisfy it in all five seeds but land far less often than with oracle comparisons, because the judge is near chance on the near-tied failures dominating early training - [Supporting source](https://arxiv.org/abs/2605.06375) ## Source-stated The paper says comparisons do not close the gap to PPO with scalar rewards on HalfCheetah. > Comparisons do not close the gap to PPO with scalar rewards on HalfCheetah. - [Supporting source](https://arxiv.org/abs/2605.06375) ## Source and contributor context Yu, Hao · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Reinforcement learning Training by trial and error, in which a system is rewarded for the outcomes its designers want. - [Definition reference](https://developers.google.com/machine-learning/glossary#reinforcement-learning-rl) ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## TIME TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services. - [Explore this context](/entities/time) - [Reference](https://time.com/) ## Editorial provenance Version 1. Publication ID: 9d6c1dd0-b1ee-4672-8135-06bd48c4ef53. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T12:34:10.445Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 07910f652aded99fd0146609d228998832041b852d7c89e6496d890bd7911bf0. Source text hash: 840c9e01a25a716fe5c21509f6625be7b44077aece4ac6d187e31cad24106ea1. ### uncertain — 2026-11-09T12:34:10.445Z — anthropic/claude-opus-5.5 Publication: 9d6c1dd0-b1ee-4672-8135-06bd48c4ef53 attribution — pass: The reading correctly attributes the paper to Hao Yu on arXiv. It frames the findings as the paper's own claims, using phrases such as 'the paper says' and 'the abstract reports'. > Authors:Hao Yu summary — uncertain: The reading leaves out the author's core motivation: scalar rewards are often impossible to write down, while judging which outcome is better is easy. That is the stake that makes the work matter to a general reader. 'Landing far less often' is not explained, so a reader cannot tell what landing means here. Smallest repair: open with the motivating sentence and say plainly what 'land' measures. Central terms such as reward-to-go, PPO, HalfCheetah and oracle comparisons are kept exactly and should get term-library tooltips. The title and question are otherwise faithful. > for many tasks no formula captures what we want, while judging which of two outcomes is better is easy date — pass: originalPublished is 2026-05-07, which matches the v1 submission date. It is not the 4 Nov 2026 revision date or the feed date. > Submitted on 7 May 2026 (v1), last revised 4 Nov 2026 (this version, v2) support — uncertain: All seven claims match their excerpts verbatim. However, the closing sentence says the results make the judge's ability to distinguish close outcomes 'central to whether comparison-based training works'. That is an unlabeled interpretation that goes beyond the abstract: the abstract says policies satisfied the requirement in all five seeds and attributes only the landing gap to the judge. Smallest repair: label the sentence as Escape Meridian interpretation, or narrow it to the landing-rate gap the paper explains. > because the judge is near chance on the near-tied failures dominating early training media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most relevant way to access the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/72f879f2eb1a1b893fc3841611b61228e0222da3cf46674233daab3ad893fa5c/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-ordinal-credit-assignment-learning-from-comparisons-with-72f879f2.json)