Li, Yu, Zhou, Yufa, Lan, Tian · Oct 29, 2026
On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning
Markdown · JSON · Original source
Authors Yu Li, Yufa Zhou and Tian Lan argue that training-inference mismatch in has two sources: differences in engine numerics and stale weights during asynchronous sampling. Because the sources behave differently, a correction that works for one may fail for the other. The authors describe numerical discrepancy as approximately constant, biasing updates by a nearly fixed amount. By contrast, policy lag arises from the trainer’s updates: the resulting bias enlarges the next lag, creating a feedback loop. They report that importance weighting cuts this loop only while the lag ratio stays concentrated; on Qwen3-1.7B, it fails when the ratio’s effective sample size falls below one half. Their discrepancy-lag correction (DLC) centers numerical discrepancy, clips the lag ratio against sampler weights and resynchronizes the sampler when effective sample size falls below one half. In tests with Qwen3-0.6B and Qwen3-1.7B, the authors report that DLC kept the KL divergence from on-policy training within twice the KL between two independent on-policy trainings. They also report that resynchronization prevented collapses missed by per-token corrections and a KL measure on the sampler’s prefixes.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 29, 2026 (2026-10-29).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-11T02:25:55.827Z. First observed: 2026-11-02T05:03:16.864Z. Retrieved: 2026-11-03T04:16:53.795Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The supplied evidence is the abstract; it does not include detailed methods, experimental conditions, or results beyond the reported claims. The reported evaluations name Qwen3-0.6B and Qwen3-1.7B; the abstract does not establish performance on other models or conditions.
Open question
How well does DLC perform beyond these two models and the conditions evaluated in the paper?
Source-stated
The authors identify engine numerics and stale weights in asynchronous sampling as two sources of training-inference mismatch, and say corrections suited to one source may fail for the other.
The mismatch has two sources, the numerics of the engine and the stale weights of asynchronous sampling, and a correction that stabilizes training under one source can fail under the other.
Source-stated
The authors characterize numerical discrepancy as approximately constant and policy lag as a feedback loop driven by the trainer’s own updates.
The numerical discrepancy stays approximately constant, so it biases the updates by a nearly fixed amount, and centering the score at each prefix removes its leading bias. The policy lag is produced by the trainer's own updates
Source-stated
The abstract reports that importance weighting fails on Qwen3-1.7B at the step when the lag ratio’s effective sample size falls below one half.
on Qwen3-1.7B it fails at the step where the effective sample size of the ratio falls below one half.
Source-stated
The authors report that DLC kept the measured KL divergence within twice the comparison between two independent on-policy trainings on Qwen3-0.6B and Qwen3-1.7B, and that resynchronization prevented collapses missed by other measures.
Under a stale sampler, DLC keeps the KL divergence from on-policy training within twice the KL between two independent on-policy trainings on Qwen3-0.6B and Qwen3-1.7B, and its resynchronization prevents collapses
Source and contributor context
Li, Yu, Zhou, Yufa, Lan, Tian · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Reinforcement learning
Training by trial and error, in which a system is rewarded for the outcomes its designers want.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 1b3d1c2b-aee1-4dd1-8419-6871f1ef1502.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T02:25:55.827Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 5ecb0aab8046e02dbc9c230215e7e728fd064d7f7ea7ff87a6088420b23af96b. Source text hash: 496f070b09cfabbde06a6efe0c7a677e9e233a40e0b1ed51ada3bbdd868995ab.
Revision and source-check history
Version 1 · 2026-11-11T02:25:55.827Z
On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning
Authors Yu Li, Yufa Zhou and Tian Lan argue that training-inference mismatch in has two sources: differences in engine numerics and stale weights during asynchronous sampling. Because the sources behave differently, a correction that works for one may fail for the other. The authors describe numerical discrepancy as approximately constant, biasing updates by a nearly fixed amount. By contrast, policy lag arises from the trainer’s updates: the resulting bias enlarges the next lag, creating a feedback loop. They report that importance weighting cuts this loop only while the lag ratio stays concentrated; on Qwen3-1.7B, it fails when the ratio’s effective sample size falls below one half. Their discrepancy-lag correction (DLC) centers numerical discrepancy, clips the lag ratio against sampler weights and resynchronizes the sampler when effective sample size falls below one half. In tests with Qwen3-0.6B and Qwen3-1.7B, the authors report that DLC kept the KL divergence from on-policy training within twice the KL between two independent on-policy trainings. They also report that resynchronization prevented collapses missed by per-token corrections and a KL measure on the sampler’s prefixes.
Original source
cs.AI · Article
Source details
Original source date: Oct 29, 2026 (2026-10-29).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-11T02:25:55.827Z. First observed: 2026-11-02T05:03:16.864Z. Retrieved: 2026-11-03T04:16:53.795Z.
Limits
The supplied evidence is the abstract; it does not include detailed methods, experimental conditions, or results beyond the reported claims. The reported evaluations name Qwen3-0.6B and Qwen3-1.7B; the abstract does not establish performance on other models or conditions.
Open question
How well does DLC perform beyond these two models and the conditions evaluated in the paper?
Source-stated
The authors identify engine numerics and stale weights in asynchronous sampling as two sources of training-inference mismatch, and say corrections suited to one source may fail for the other.
The mismatch has two sources, the numerics of the engine and the stale weights of asynchronous sampling, and a correction that stabilizes training under one source can fail under the other.
Source-stated
The authors characterize numerical discrepancy as approximately constant and policy lag as a feedback loop driven by the trainer’s own updates.
The numerical discrepancy stays approximately constant, so it biases the updates by a nearly fixed amount, and centering the score at each prefix removes its leading bias. The policy lag is produced by the trainer's own updates
Source-stated
The abstract reports that importance weighting fails on Qwen3-1.7B at the step when the lag ratio’s effective sample size falls below one half.
on Qwen3-1.7B it fails at the step where the effective sample size of the ratio falls below one half.
Source-stated
The authors report that DLC kept the measured KL divergence within twice the comparison between two independent on-policy trainings on Qwen3-0.6B and Qwen3-1.7B, and that resynchronization prevented collapses missed by other measures.
Under a stale sampler, DLC keeps the KL divergence from on-policy training within twice the KL between two independent on-policy trainings on Qwen3-0.6B and Qwen3-1.7B, and its resynchronization prevents collapses
Source and contributor context
Li, Yu, Zhou, Yufa, Lan, Tian · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Reinforcement learning
Training by trial and error, in which a system is rewarded for the outcomes its designers want.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 1b3d1c2b-aee1-4dd1-8419-6871f1ef1502.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T02:25:55.827Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 5ecb0aab8046e02dbc9c230215e7e728fd064d7f7ea7ff87a6088420b23af96b. Source text hash: 496f070b09cfabbde06a6efe0c7a677e9e233a40e0b1ed51ada3bbdd868995ab.
uncertain · 2026-11-11T02:25:55.827Z · anthropic/claude-opus-5.5
Publication: 1b3d1c2b-aee1-4dd1-8419-6871f1ef1502
attribution — pass: The reading correctly attributes the work to Yu Li, Yufa Zhou and Tian Lan. It frames the mechanisms and results as the authors' descriptions and reports rather than as established facts.
Authors:Yu Li, Yufa Zhou, Tian Lan
summary — uncertain: The summary never explains what training-inference mismatch is. The abstract says an inference engine samples responses while a separate trainer computes gradients, which makes the training data off-policy. Without that, a general reader cannot follow the central problem. On-policy, importance weighting, effective sample size and KL divergence are also left unexplained; they need tooltips. The reading also does not say why the work matters, namely training stability and collapse prevention. Smallest repair: add one plain sentence on the engine/trainer split and why off-policy data destabilizes training.
In practice, an inference engine samples the responses and a separate trainer computes the gradients, and the \textbf{training-inference mismatch} between the two makes the training data off-policy.
date — pass: originalPublished 2026-10-29 matches the v1 submission date on arXiv.
[Submitted on 29 Oct 2026]
support — uncertain: Most claims track the abstract closely. However, the summary paragraph and claim 4 drop the condition that the DLC KL result was obtained under a stale sampler. That condition scopes the result. Smallest repair: insert 'Under a stale sampler,' before the DLC KL result. The question about generalization beyond the two models follows reasonably from the evidence.
Under a stale sampler, DLC keeps the KL divergence from on-policy training within twice the KL between two independent on-policy trainings on Qwen3-0.6B and Qwen3-1.7B
media — pass: The selected media is the verified direct arXiv abstract page for this paper, the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A language model with many learned numerical settings, used to process and generate language.
Definition referenceTraining by trial and error, in which a system is rewarded for the outcomes its designers want.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referenceTraining by trial and error, in which a system is rewarded for the outcomes its designers want.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv