# arXiv cs.AI: KL for a KL: On-Policy Distillation with Control Variate Baseline [Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-kl-for-a-kl-on-policy-distillation-with-control-628befb3) The authors of “KL for a KL” argue that high gradient variance makes On-Policy Distillation (OPD) unstable, and propose vOPD to reduce that variance without adding a separate critic or inference. They treat OPD as policy-gradient RL and use a value function: the per-token negative reverse KL divergence between student and teacher, which they say is available from the existing forward pass. The authors say subtracting this value function as a detached baseline keeps the gradient unbiased while reducing variance. They contrast vOPD’s lightweight single-sample estimator with approaches that calculate the full token-level reverse KL over the entire vocabulary or restrict it to a top-k support. They also report that a top-k approximation of the baseline lowers cost without compromising performance. Across mathematics, science, and instruction-following benchmarks, the authors report that vOPD outperforms base OPD and matches a more expensive full-vocabulary method. The abstract does not give benchmark results or implementation costs in detail, so the scope of those comparisons cannot be assessed from this evidence alone. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: May 8, 2026 (2026-05-08). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-09T12:19:00.131Z. First observed: 2026-11-05T05:09:34.361Z. Retrieved: 2026-11-09T12:18:09.172Z. - [KL for a KL: On-Policy Distillation with Control Variate Baseline](https://arxiv.org/abs/2605.07865) ## Limits The supplied evidence is the paper abstract; it does not include benchmark-by-benchmark results or details of how the comparisons were measured. The abstract does not quantify the implementation costs of the full-vocabulary method or the top-k approximation. ## Open question How do the reported comparisons vary across the mathematics, science, and instruction-following benchmarks, and what implementation costs do the authors measure? ## Source-stated The authors identify high gradient variance in OPD's single-sample Monte Carlo estimator as a source of practical instability. > OPD remains unstable in practice due to the high gradient variance of its single-sample Monte Carlo estimator - [Supporting source](https://arxiv.org/abs/2605.07865) ## Source-stated The authors propose vOPD, which uses a value function as a control variate baseline; they identify that value function as the per-token negative reverse KL divergence between student and teacher. > the per-token negative reverse KL divergence between student and teacher, available directly from the existing forward pass with no additional critic or inference. - [Supporting source](https://arxiv.org/abs/2605.07865) ## Source-stated The authors say subtracting the value function as a detached baseline preserves an unbiased gradient while reducing variance. > subtracts the value function as a detached baseline to keep the gradient unbiased while reducing variance. - [Supporting source](https://arxiv.org/abs/2605.07865) ## Source-stated The authors report that vOPD outperforms base OPD and matches the most expensive full-vocabulary method across the stated benchmark areas. > Across mathematics, science, and instruction-following benchmarks, vOPD consistently outperforms base OPD and matches the most expensive full-vocabulary method - [Supporting source](https://arxiv.org/abs/2605.07865) ## Source and contributor context Oh, Minjae, Song, Sangjun, Choi, Gyubin, Choi, Yunho, Jo, Yohan · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Reinforcement learning Training by trial and error, in which a system is rewarded for the outcomes its designers want. - [Definition reference](https://developers.google.com/machine-learning/glossary#reinforcement-learning-rl) ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 3e2cf441-e7a9-4c8f-84ee-9eb10b450176. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T12:19:00.131Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: ffba6573c8449b35be466259e3b3b80954293670bb01d896f97fb7c37fa6f96f. Source text hash: adbf87cd84bc4803303983c85a69896d884721c7e35186026ff4c2d2b9047860. ## Revision and source-check history ### Version 1 — 2026-11-09T12:19:00.131Z The authors of “KL for a KL” argue that high gradient variance makes On-Policy Distillation (OPD) unstable, and propose vOPD to reduce that variance without adding a separate critic or inference. They treat OPD as policy-gradient RL and use a value function: the per-token negative reverse KL divergence between student and teacher, which they say is available from the existing forward pass. The authors say subtracting this value function as a detached baseline keeps the gradient unbiased while reducing variance. They contrast vOPD’s lightweight single-sample estimator with approaches that calculate the full token-level reverse KL over the entire vocabulary or restrict it to a top-k support. They also report that a top-k approximation of the baseline lowers cost without compromising performance. Across mathematics, science, and instruction-following benchmarks, the authors report that vOPD outperforms base OPD and matches a more expensive full-vocabulary method. The abstract does not give benchmark results or implementation costs in detail, so the scope of those comparisons cannot be assessed from this evidence alone. ## Original source arXiv cs.AI · Article Original source date: May 8, 2026 (2026-05-08). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-09T12:19:00.131Z. First observed: 2026-11-05T05:09:34.361Z. Retrieved: 2026-11-09T12:18:09.172Z. - [KL for a KL: On-Policy Distillation with Control Variate Baseline](https://arxiv.org/abs/2605.07865) ## Limits The supplied evidence is the paper abstract; it does not include benchmark-by-benchmark results or details of how the comparisons were measured. The abstract does not quantify the implementation costs of the full-vocabulary method or the top-k approximation. ## Open question How do the reported comparisons vary across the mathematics, science, and instruction-following benchmarks, and what implementation costs do the authors measure? ## Source-stated The authors identify high gradient variance in OPD's single-sample Monte Carlo estimator as a source of practical instability. > OPD remains unstable in practice due to the high gradient variance of its single-sample Monte Carlo estimator - [Supporting source](https://arxiv.org/abs/2605.07865) ## Source-stated The authors propose vOPD, which uses a value function as a control variate baseline; they identify that value function as the per-token negative reverse KL divergence between student and teacher. > the per-token negative reverse KL divergence between student and teacher, available directly from the existing forward pass with no additional critic or inference. - [Supporting source](https://arxiv.org/abs/2605.07865) ## Source-stated The authors say subtracting the value function as a detached baseline preserves an unbiased gradient while reducing variance. > subtracts the value function as a detached baseline to keep the gradient unbiased while reducing variance. - [Supporting source](https://arxiv.org/abs/2605.07865) ## Source-stated The authors report that vOPD outperforms base OPD and matches the most expensive full-vocabulary method across the stated benchmark areas. > Across mathematics, science, and instruction-following benchmarks, vOPD consistently outperforms base OPD and matches the most expensive full-vocabulary method - [Supporting source](https://arxiv.org/abs/2605.07865) ## Source and contributor context Oh, Minjae, Song, Sangjun, Choi, Gyubin, Choi, Yunho, Jo, Yohan · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Reinforcement learning Training by trial and error, in which a system is rewarded for the outcomes its designers want. - [Definition reference](https://developers.google.com/machine-learning/glossary#reinforcement-learning-rl) ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 3e2cf441-e7a9-4c8f-84ee-9eb10b450176. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T12:19:00.131Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: ffba6573c8449b35be466259e3b3b80954293670bb01d896f97fb7c37fa6f96f. Source text hash: adbf87cd84bc4803303983c85a69896d884721c7e35186026ff4c2d2b9047860. ### uncertain — 2026-11-09T12:19:00.131Z — anthropic/claude-opus-5.5 Publication: 3e2cf441-e7a9-4c8f-84ee-9eb10b450176 attribution — pass: The claims are consistently attributed to the paper's authors as proposals, statements or reported results. The title correctly names the arXiv paper, and the 'arXiv cs.AI' label matches a listed subject, although the primary category is cs.LG. > Authors:Minjae Oh, Sangjun Song, Gyubin Choi, Yunho Choi, Yohan Jo summary — uncertain: The summary is faithful, but its central terms remain unexplained for a general reader: OPD, student/teacher, policy-gradient RL, control variate baseline, value function, reverse KL divergence, forward pass and top-k. It also omits the abstract's framing that OPD is a dominant post-training method for LLMs, which is why stability matters. Smallest repair: add one plain sentence saying OPD trains a student model to match a teacher model on the student's own outputs and is widely used, add tooltips for these terms, and explain that the existing fixes are either costly (full vocabulary) or biased (top-k). It also softens 'the most expensive' to 'a more expensive' and drops 'consistently'. > On-Policy Distillation (OPD) has emerged as a dominant post-training paradigm for Large Language Models, especially in reasoning domains. date — pass: The originalPublished date of 2026-05-08 matches the v1 submission. It is not the 4 Nov 2026 revision or the feed date. > \[Submitted on 8 May 2026 (v1), last revised 4 Nov 2026 (this version, v2)\] support — pass: Each claim has a matching verbatim excerpt and remains attributed to the authors. The limitations and the question accurately note that the abstract gives no detailed results or cost figures. No unsupported benefits are implied. Benchmark results are reported, not independently verified. > vOPD instead preserves the lightweight single-sample estimator and subtracts the value function as a detached baseline to keep the gradient unbiased while reducing variance. media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/628befb3bf4fdf37ced10b507c5cf0d9025cf30f6128cce2cdd893d05bb02c4e/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-kl-for-a-kl-on-policy-distillation-with-control-628befb3.json)