arXiv cs.AI: KL for a KL: On-Policy Distillation with Control Variate Baseline

arXiv cs.AI: KL for a KL: On-Policy Distillation with Control Variate Baseline

The authors of “KL for a KL” argue that high gradient variance makes On-Policy Distillation (OPD) unstable, and propose vOPD to reduce that variance without adding a separate critic or inference. They treat OPD as policy-gradient RL and use a value function: the per-token negative reverse KL divergence between student and teacher, which they say is available from the existing forward pass. The authors say subtracting this value function as a detached baseline keeps the gradient unbiased while reducing variance. They contrast vOPD’s lightweight single-sample estimator with approaches that calculate the full token-level reverse KL over the entire vocabulary or restrict it to a top-k support. They also report that a top-k approximation of the baseline lowers cost without compromising performance. Across mathematics, science, and instruction-following benchmarks, the authors report that vOPD outperforms base OPD and matches a more expensive full-vocabulary method. The abstract does not give benchmark results or implementation costs in detail, so the scope of those comparisons cannot be assessed from this evidence alone.

Publication: 3e2cf441-e7a9-4c8f-84ee-9eb10b450176

attribution — pass: The claims are consistently attributed to the paper's authors as proposals, statements or reported results. The title correctly names the arXiv paper, and the 'arXiv cs.AI' label matches a listed subject, although the primary category is cs.LG.

Authors:Minjae Oh, Sangjun Song, Gyubin Choi, Yunho Choi, Yohan Jo

summary — uncertain: The summary is faithful, but its central terms remain unexplained for a general reader: OPD, student/teacher, policy-gradient RL, control variate baseline, value function, reverse KL divergence, forward pass and top-k. It also omits the abstract's framing that OPD is a dominant post-training method for LLMs, which is why stability matters. Smallest repair: add one plain sentence saying OPD trains a student model to match a teacher model on the student's own outputs and is widely used, add tooltips for these terms, and explain that the existing fixes are either costly (full vocabulary) or biased (top-k). It also softens 'the most expensive' to 'a more expensive' and drops 'consistently'.

On-Policy Distillation (OPD) has emerged as a dominant post-training paradigm for Large Language Models, especially in reasoning domains.

date — pass: The originalPublished date of 2026-05-08 matches the v1 submission. It is not the 4 Nov 2026 revision or the feed date.

[Submitted on 8 May 2026 (v1), last revised 4 Nov 2026 (this version, v2)]

support — pass: Each claim has a matching verbatim excerpt and remains attributed to the authors. The limitations and the question accurately note that the abstract gives no detailed results or cost figures. No unsupported benefits are implied. Benchmark results are reported, not independently verified.

vOPD instead preserves the lightweight single-sample estimator and subtracts the value function as a detached baseline to keep the gradient unbiased while reducing variance.

media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.