Yu, Hao · May 7, 2026
arXiv: Ordinal Credit Assignment: Learning from comparisons, with a judge’s limits
Markdown · JSON · Original source
Hao Yu’s paper studies multi-step using comparisons between outcomes rather than a reward function or reward model. It proposes crediting each action by how its reward-to-go compares with other trajectories still running at the same step. In deterministic environments, the expected update ranks actions by true value; with stochastic outcomes, it favors actions that win more often, not necessarily those with higher mean. In a pre-registered study, comparisons cost no detectable performance relative to scalar returns at the same granularity, and learning a reward model from those comparisons did not do better. With real judges, comparisons were 7–10 points more accurate than pointwise scores. But when training with an LLM judge on a requirement omitted by a hand-written reward, policies satisfied the requirement in all five seeds while landing far less often than with oracle comparisons. The paper attributes that gap to the judge being near chance on near-tied failures that dominate early training. The abstract also reports that comparisons do not close the gap to PPO with scalar rewards on HalfCheetah. These results make the judge’s ability to distinguish close outcomes central to whether comparison-based training works in the reported setting.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: May 7, 2026 (2026-05-07).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T12:34:10.445Z. First observed: 2026-11-05T05:00:23.346Z. Retrieved: 2026-11-09T12:33:01.777Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The supplied evidence is the abstract, so it does not provide the full experimental setup or detail behind the reported results. The reported study covers six environments with ten seeds per claim; the -judge training result covers five seeds. The abstract reports that the judge is near chance on near-tied failures, but does not establish whether improving that accuracy would reduce the performance gap.
Open question
Would a judge that is more accurate on near-tied failures reduce the gap between -judge and oracle-comparison training?
Source-stated
The paper studies multi-step reinforcement learning from comparisons, without a reward function or reward model.
We study reinforcement learning that receives only such comparisons - no reward function, no reward model - in multi-step tasks.
Source-stated
Its proposed update credits actions by comparing reward-to-go against other trajectories still running at that time step.
each action is credited by how its reward-to-go compares with those of other trajectories at the same time step
Source-stated
The paper says the update ranks actions by true value in deterministic environments, but in stochastic outcomes favors actions that win more often over those with higher mean.
under stochastic outcomes it prefers actions that win more often rather than those with higher mean
Source-stated
In the reported study, comparisons had no detectable performance cost relative to scalar returns at the same granularity, and learning a reward model from those comparisons was not better.
comparisons cost no detectable performance relative to scalar returns at the same granularity, learning a reward model from the same comparisons is not better
Source-stated
With real LLM judges, comparisons were 7–10 points more accurate than pointwise scores.
With real LLM judges, comparisons are 7-10 points more accurate than pointwise scores
Source-stated
In training on a requirement omitted by the hand-written reward, policies satisfied it in all five seeds but landed far less often than with oracle comparisons; the paper attributes the gap to the judge’s poor performance on near-tied failures.
policies satisfy it in all five seeds but land far less often than with oracle comparisons, because the judge is near chance on the near-tied failures dominating early training
Source-stated
The paper says comparisons do not close the gap to PPO with scalar rewards on HalfCheetah.
Comparisons do not close the gap to PPO with scalar rewards on HalfCheetah.
Source and contributor context
Yu, Hao · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Reinforcement learning
Training by trial and error, in which a system is rewarded for the outcomes its designers want.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 9d6c1dd0-b1ee-4672-8135-06bd48c4ef53.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T12:34:10.445Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 07910f652aded99fd0146609d228998832041b852d7c89e6496d890bd7911bf0. Source text hash: 840c9e01a25a716fe5c21509f6625be7b44077aece4ac6d187e31cad24106ea1.
Revision and source-check history
Version 1 · 2026-11-09T12:34:10.445Z
arXiv: Ordinal Credit Assignment: Learning from comparisons, with a judge’s limits
Hao Yu’s paper studies multi-step using comparisons between outcomes rather than a reward function or reward model. It proposes crediting each action by how its reward-to-go compares with other trajectories still running at the same step. In deterministic environments, the expected update ranks actions by true value; with stochastic outcomes, it favors actions that win more often, not necessarily those with higher mean. In a pre-registered study, comparisons cost no detectable performance relative to scalar returns at the same granularity, and learning a reward model from those comparisons did not do better. With real judges, comparisons were 7–10 points more accurate than pointwise scores. But when training with an LLM judge on a requirement omitted by a hand-written reward, policies satisfied the requirement in all five seeds while landing far less often than with oracle comparisons. The paper attributes that gap to the judge being near chance on near-tied failures that dominate early training. The abstract also reports that comparisons do not close the gap to PPO with scalar rewards on HalfCheetah. These results make the judge’s ability to distinguish close outcomes central to whether comparison-based training works in the reported setting.
Original source
cs.AI · Article
Source details
Original source date: May 7, 2026 (2026-05-07).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T12:34:10.445Z. First observed: 2026-11-05T05:00:23.346Z. Retrieved: 2026-11-09T12:33:01.777Z.
Limits
The supplied evidence is the abstract, so it does not provide the full experimental setup or detail behind the reported results. The reported study covers six environments with ten seeds per claim; the -judge training result covers five seeds. The abstract reports that the judge is near chance on near-tied failures, but does not establish whether improving that accuracy would reduce the performance gap.
Open question
Would a judge that is more accurate on near-tied failures reduce the gap between -judge and oracle-comparison training?
Source-stated
The paper studies multi-step reinforcement learning from comparisons, without a reward function or reward model.
We study reinforcement learning that receives only such comparisons - no reward function, no reward model - in multi-step tasks.
Source-stated
Its proposed update credits actions by comparing reward-to-go against other trajectories still running at that time step.
each action is credited by how its reward-to-go compares with those of other trajectories at the same time step
Source-stated
The paper says the update ranks actions by true value in deterministic environments, but in stochastic outcomes favors actions that win more often over those with higher mean.
under stochastic outcomes it prefers actions that win more often rather than those with higher mean
Source-stated
In the reported study, comparisons had no detectable performance cost relative to scalar returns at the same granularity, and learning a reward model from those comparisons was not better.
comparisons cost no detectable performance relative to scalar returns at the same granularity, learning a reward model from the same comparisons is not better
Source-stated
With real LLM judges, comparisons were 7–10 points more accurate than pointwise scores.
With real LLM judges, comparisons are 7-10 points more accurate than pointwise scores
Source-stated
In training on a requirement omitted by the hand-written reward, policies satisfied it in all five seeds but landed far less often than with oracle comparisons; the paper attributes the gap to the judge’s poor performance on near-tied failures.
policies satisfy it in all five seeds but land far less often than with oracle comparisons, because the judge is near chance on the near-tied failures dominating early training
Source-stated
The paper says comparisons do not close the gap to PPO with scalar rewards on HalfCheetah.
Comparisons do not close the gap to PPO with scalar rewards on HalfCheetah.
Source and contributor context
Yu, Hao · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Reinforcement learning
Training by trial and error, in which a system is rewarded for the outcomes its designers want.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 9d6c1dd0-b1ee-4672-8135-06bd48c4ef53.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T12:34:10.445Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 07910f652aded99fd0146609d228998832041b852d7c89e6496d890bd7911bf0. Source text hash: 840c9e01a25a716fe5c21509f6625be7b44077aece4ac6d187e31cad24106ea1.
uncertain · 2026-11-09T12:34:10.445Z · anthropic/claude-opus-5.5
Publication: 9d6c1dd0-b1ee-4672-8135-06bd48c4ef53
attribution — pass: The reading correctly attributes the paper to Hao Yu on arXiv. It frames the findings as the paper's own claims, using phrases such as 'the paper says' and 'the abstract reports'.
Authors:Hao Yu
summary — uncertain: The reading leaves out the author's core motivation: scalar rewards are often impossible to write down, while judging which outcome is better is easy. That is the stake that makes the work matter to a general reader. 'Landing far less often' is not explained, so a reader cannot tell what landing means here. Smallest repair: open with the motivating sentence and say plainly what 'land' measures. Central terms such as reward-to-go, PPO, HalfCheetah and oracle comparisons are kept exactly and should get term-library tooltips. The title and question are otherwise faithful.
for many tasks no formula captures what we want, while judging which of two outcomes is better is easy
date — pass: originalPublished is 2026-05-07, which matches the v1 submission date. It is not the 4 Nov 2026 revision date or the feed date.
Submitted on 7 May 2026 (v1), last revised 4 Nov 2026 (this version, v2)
support — uncertain: All seven claims match their excerpts verbatim. However, the closing sentence says the results make the judge's ability to distinguish close outcomes 'central to whether comparison-based training works'. That is an unlabeled interpretation that goes beyond the abstract: the abstract says policies satisfied the requirement in all five seeds and attributes only the landing gap to the judge. Smallest repair: label the sentence as Escape Meridian interpretation, or narrow it to the landing-rate gap the paper explains.
because the judge is near chance on the near-tied failures dominating early training
media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most relevant way to access the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Training by trial and error, in which a system is rewarded for the outcomes its designers want.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEA language model with many learned numerical settings, used to process and generate language.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceTraining by trial and error, in which a system is rewarded for the outcomes its designers want.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEA language model with many learned numerical settings, used to process and generate language.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition reference