arXiv cs.AI: Proposing Better Rollouts for On-Policy Distillation

arXiv cs.AI: Proposing Better Rollouts for On-Policy Distillation

In “Proposing Better Rollouts for On-Policy Distillation,” Sirui Chen and coauthors argue that on-policy distillation can go wrong when the student’s own sampled trajectories are used as rollouts in every training round. They say the teacher may give noisy or misleading supervision on trajectories that are less reachable by the teacher, encouraging patterns such as repetition that can persist or grow in later rounds. The authors propose Mixture Proposal On-Policy Distillation (MPOPD), which combines the student and teacher policies into a rollout distribution and approximates it with a proposal policy. They report consistent gains over on-policy distillation and recent variants on 16 benchmarks. On math reasoning, they report a 6.91 improvement in average accuracy over OPD for Qwen3-4B-Base, alongside a 41% reduction in average generation length. These are reported benchmark results; the abstract does not establish how well the method transfers to other tasks or settings. An open question is whether MPOPD’s reported gains hold beyond the benchmarks and model setting described in the abstract.

Publication: 520d9fd4-4a40-41f5-82ec-2298b9ad082a

attribution — pass: The draft correctly attributes the paper to Sirui Chen and coauthors on arXiv. It keeps the arguments and benchmark results framed as the authors' reported claims.

Authors:Sirui Chen, Yunzhe Qi, Haobo Xu, Jiaru Zou, Mengting Ai, Zihao Li, Jingrui He

summary — uncertain: The summary is faithful, but a non-specialist could not follow it. It never explains on-policy distillation: a smaller student model learns from its own sampled outputs while a teacher model supervises. It also never says that OPD is a language-model post-training method or what rollouts, policy and trajectories mean. It omits the authors' central thesis that the rollout policy itself should be an optimization target, and the single-forward-pass point. The closing open-question sentence repeats the question. Smallest repair: add a one-sentence plain explanation of OPD and rollouts, and state the authors' thesis.

On-policy distillation (OPD) improves language-model post-training by letting the student learn from its own sampled trajectories under teacher supervision

date — pass: originalPublished is 2026-10-27, which matches the v1 submission date. It is not the date of the later v2 revision or the feed date.

Submitted on 27 Oct 2026 (v1), last revised 28 Oct 2026 (this version, v2)

support — pass: Every claim matches the abstract, including the mismatch mechanism, the geometric mixture, the 16 benchmarks, the 6.91 accuracy gain and the 41% length reduction. The limitations and the question are appropriately conditional and do not overstate real-world reliability.

On math reasoning, our method improves average accuracy by 6.91 over OPD on Qwen3-4B-Base while reducing average generation length by 41%

media — pass: The selected media is the verified arXiv abstract page, which is the direct original source.