Singh, Vikash, Chen, Weicong, Ganguly, Debargha, Zhang, Yanyan, Hariri, Mohsen, Sankar, Sreehari, Yang, Wang, Song, Chaoda, Nemecek, Alexander, Zafar, Osama, Shao, Yuting, Ye, Kai, Zhang, Biyao, Li, Xinpeng, Wang, Shouren, Ayday, Erman, Ma, Jing, Kuppannagari, Sanmukh Rao, Chaudhary, Vipin · Nov 10, 2026
Are Dense Rewards Worth Their Tokens in GRPO?
Markdown · JSON · Original source
The paper’s authors argue that dense process rewards can look better when compared per training update, but lose their advantage when the compute they consume is counted. In their setting, continuations used to calculate a Monte-Carlo reward account for 91% of generated tokens. Across 1,668 runs on six models, the authors report that matching compute reverses the ranking: on Qwen2.5-1.5B, the Monte-Carlo reward trails standard GRPO by 14.3 percentage points at equal tokens and 7.8 points at equal training . None of four dense-reward variants outperforms GRPO even with as much training time. A per-token gradient test ranks the rewards in the same order as training, but overestimates them by one to two points—up to the size of the gains it is meant to detect. The authors also report gains from training on prompts whose predicted success is nearest 1/2: 1.7 points on Qwen3-0.6B and 0.8 on Qwen2.5-1.5B. Those gains replicated on fresh seeds and unseen problems, but both came early in training; standard GRPO with twice the initial learning rate came within half a point. Their argument is that interventions should be compared at matched compute, since that comparison can overturn apparent gains as well as confirm others.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 10, 2026 (2026-11-10).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-12T04:31:53.037Z. First observed: 2026-11-11T05:09:23.108Z. Retrieved: 2026-11-12T04:20:28.980Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The reported results are from the paper’s setting and its 1,668 runs on six models; the abstract does not establish whether the findings generalize beyond those tests. The reported prompt-selection gains came early in training, and standard GRPO with twice the initial learning rate came within half a point of them.
Open question
Do these matched-compute results hold beyond the six models and preregistered tests reported in the paper?
Source-stated
The authors report that continuations used for the Monte-Carlo reward consumed 91% of generated tokens in their setting.
in our setting, the continuations alone take 91% of all generated tokens
Source-stated
Across 1,668 runs on six models, matching compute reversed the ranking of dense rewards and GRPO.
Across 1,668 runs on six models, matching compute reverses the ranking.
Source-stated
On Qwen2.5-1.5B, the Monte-Carlo reward scored lower than standard GRPO at equal tokens and equal training time.
ends 14.3 percentage points lower at equal tokens and 7.8 points lower at equal training time
Source-stated
None of four dense-reward variants outperformed GRPO even when given as much training time.
none of four dense-reward variants outperforms GRPO even with as much training time.
Source-stated
The per-token gradient test ranked rewards as training did, but overestimated them by one to two points.
A per-token gradient test orders these rewards as training does but overestimates each by one to two points
Source-stated
The authors report matched-compute accuracy gains from training on prompts whose predicted success is nearest 1/2, with those gains replicating on fresh seeds and unseen problems.
These gains replicate on fresh seeds and unseen problems
Source-stated
Both gains appeared early in training, and standard GRPO with twice the initial learning rate came within half a point.
Both gains come early in training, and standard GRPO with twice the initial learning rate comes within half a point of them.
Source-stated
The authors argue that training interventions should be validated at matched compute because this comparison can overturn or confirm apparent gains.
We argue that training interventions should be validated at matched compute, which overturns some apparent gains and confirms others.
Source and contributor context
Singh, Vikash, Chen, Weicong, Ganguly, Debargha, Zhang, Yanyan, Hariri, Mohsen, Sankar, Sreehari, Yang, Wang, Song, Chaoda, Nemecek, Alexander, Zafar, Osama, Shao, Yuting, Ye, Kai, Zhang, Biyao, Li, Xinpeng, Wang, Shouren, Ayday, Erman, Ma, Jing, Kuppannagari, Sanmukh Rao, Chaudhary, Vipin · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 85683a5c-fba3-4f37-806c-778e0284e521.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T04:31:53.037Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: bd9ce07d999107926afbd17a659dc583bec5fc64f7a77cae984c0777ec6cf10e. Source text hash: 84d22bb323a2d1085ba2573476e539a166a9a50e0bb4b998421acdcd7e0b51e0.
Revision and source-check history
Version 1 · 2026-11-12T04:31:53.037Z
Are Dense Rewards Worth Their Tokens in GRPO?
The paper’s authors argue that dense process rewards can look better when compared per training update, but lose their advantage when the compute they consume is counted. In their setting, continuations used to calculate a Monte-Carlo reward account for 91% of generated tokens. Across 1,668 runs on six models, the authors report that matching compute reverses the ranking: on Qwen2.5-1.5B, the Monte-Carlo reward trails standard GRPO by 14.3 percentage points at equal tokens and 7.8 points at equal training . None of four dense-reward variants outperforms GRPO even with as much training time. A per-token gradient test ranks the rewards in the same order as training, but overestimates them by one to two points—up to the size of the gains it is meant to detect. The authors also report gains from training on prompts whose predicted success is nearest 1/2: 1.7 points on Qwen3-0.6B and 0.8 on Qwen2.5-1.5B. Those gains replicated on fresh seeds and unseen problems, but both came early in training; standard GRPO with twice the initial learning rate came within half a point. Their argument is that interventions should be compared at matched compute, since that comparison can overturn apparent gains as well as confirm others.
Original source
cs.AI · Article
Source details
Original source date: Nov 10, 2026 (2026-11-10).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-12T04:31:53.037Z. First observed: 2026-11-11T05:09:23.108Z. Retrieved: 2026-11-12T04:20:28.980Z.
Limits
The reported results are from the paper’s setting and its 1,668 runs on six models; the abstract does not establish whether the findings generalize beyond those tests. The reported prompt-selection gains came early in training, and standard GRPO with twice the initial learning rate came within half a point of them.
Open question
Do these matched-compute results hold beyond the six models and preregistered tests reported in the paper?
Source-stated
The authors report that continuations used for the Monte-Carlo reward consumed 91% of generated tokens in their setting.
in our setting, the continuations alone take 91% of all generated tokens
Source-stated
Across 1,668 runs on six models, matching compute reversed the ranking of dense rewards and GRPO.
Across 1,668 runs on six models, matching compute reverses the ranking.
Source-stated
On Qwen2.5-1.5B, the Monte-Carlo reward scored lower than standard GRPO at equal tokens and equal training time.
ends 14.3 percentage points lower at equal tokens and 7.8 points lower at equal training time
Source-stated
None of four dense-reward variants outperformed GRPO even when given as much training time.
none of four dense-reward variants outperforms GRPO even with as much training time.
Source-stated
The per-token gradient test ranked rewards as training did, but overestimated them by one to two points.
A per-token gradient test orders these rewards as training does but overestimates each by one to two points
Source-stated
The authors report matched-compute accuracy gains from training on prompts whose predicted success is nearest 1/2, with those gains replicating on fresh seeds and unseen problems.
These gains replicate on fresh seeds and unseen problems
Source-stated
Both gains appeared early in training, and standard GRPO with twice the initial learning rate came within half a point.
Both gains come early in training, and standard GRPO with twice the initial learning rate comes within half a point of them.
Source-stated
The authors argue that training interventions should be validated at matched compute because this comparison can overturn or confirm apparent gains.
We argue that training interventions should be validated at matched compute, which overturns some apparent gains and confirms others.
Source and contributor context
Singh, Vikash, Chen, Weicong, Ganguly, Debargha, Zhang, Yanyan, Hariri, Mohsen, Sankar, Sreehari, Yang, Wang, Song, Chaoda, Nemecek, Alexander, Zafar, Osama, Shao, Yuting, Ye, Kai, Zhang, Biyao, Li, Xinpeng, Wang, Shouren, Ayday, Erman, Ma, Jing, Kuppannagari, Sanmukh Rao, Chaudhary, Vipin · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 85683a5c-fba3-4f37-806c-778e0284e521.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T04:31:53.037Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: bd9ce07d999107926afbd17a659dc583bec5fc64f7a77cae984c0777ec6cf10e. Source text hash: 84d22bb323a2d1085ba2573476e539a166a9a50e0bb4b998421acdcd7e0b51e0.
uncertain · 2026-11-12T04:31:53.037Z · anthropic/claude-opus-5.5
Publication: 85683a5c-fba3-4f37-806c-778e0284e521
attribution — pass: The results and the matched-compute recommendation are attributed to the paper's authors throughout, using phrases such as 'the authors report' and 'their argument is'. Nothing is presented as established fact or as Escape Meridian's view.
We argue that training interventions should be validated at matched compute, which overturns some apparent gains and confirms others.
summary — uncertain: The numbers are faithful, but a non-specialist cannot tell what the paper is about. The summary never says this is about training language models to reason by rewarding correct final answers (RLVR). It does not say that GRPO gives every token the same credit, or that dense process rewards credit intermediate steps. It also never explains that the Monte-Carlo reward scores each step by sampling continuations. Smallest repair: add one opening sentence giving this context from the abstract, keeping GRPO and RLVR as tooltip terms.
RLVR trains language models to reason by rewarding correct final answers. A widely used algorithm, group-relative policy optimization (GRPO), gives every token of a solution the same credit
date — pass: originalPublished is 2026-11-10, which matches the arXiv v1 submission date. The later feed timestamp was correctly not used.
[Submitted on 10 Nov 2026]
support — pass: Every claim and figure (91%, 1,668 runs, 14.3 and 7.8 points, one-to-two-point overestimate, 1.7 and 0.8 gains, half-point learning-rate result) matches the abstract. The limitations and the question stay within the stated scope and imply no unsupported benefits or costs.
On Qwen2.5-1.5B, the Monte-Carlo reward leads standard GRPO per update, yet ends 14.3 percentage points lower at equal tokens and 7.8 points lower at equal training time
media — pass: The selected media is the verified direct link to the arXiv abstract page of the source itself, which is the most direct way to reach the paper.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv