# arXiv: Koopman Model-Based Reinforcement Learning [Canonical reading](https://escapemeridian.com/readings/arxiv-koopman-model-based-reinforcement-learning-76fb461a) Authors Wenjian Hao, Paulo C. Heredia and Shaoshuai Mou propose K-MBRL, a method for controlling nonlinear robotic systems with unknown dynamics. It combines a learned model with a critic, which estimates future cost, to improve a control policy. The authors say representing dynamics with linear Koopman dynamics makes policy-gradient computation a constant matrix product, while using a one-step predicted state avoids repeated long-horizon model predictions. The critic retains information about future cost. The resulting policy acts through feedforward evaluation, without online optimization. The authors report convergence guarantees under standard stochastic approximation assumptions. They say simulations on HalfCheetah and quadruped locomotion tasks show strong sample efficiency and improved performance against classical reinforcement-learning baselines. They also report testing on a physical quadruped: it rose onto two rounded rear feet, balanced during bipedal walking, and returned to the ground. The physical-robot validation extends beyond simulation, but the abstract describes only this sequence of maneuvers. The authors use a pendulum task to assess policy-gradient approximation accuracy and examine components and hyperparameters. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv eess.SY · Article Original source date: Nov 10, 2026 (2026-11-10). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-11T22:33:42.728Z. First observed: 2026-11-11T05:05:08.537Z. Retrieved: 2026-11-11T22:32:41.713Z. - [Koopman Model-Based Reinforcement Learning](https://arxiv.org/abs/2611.14186) ## Limits The evidence is the arXiv abstract; it reports qualitative comparisons but gives no numerical performance results in the supplied text. The physical quadruped validation described here covers a specific sequence of maneuvers; the supplied abstract does not report performance beyond it. Technical terms to consider for the term library: Koopman model-based reinforcement learning, Koopman operator, critic, policy gradient, feedforward evaluation, and stochastic approximation. ## Open question How does K-MBRL perform on the physical quadruped beyond the described sequence of maneuvers? ## Source-stated The authors propose K-MBRL for controlling nonlinear robotic systems with unknown dynamics, using a learned policy objective that combines immediate cost with estimated future cost. > We propose a Koopman model-based reinforcement learning (K-MBRL) algorithm for controlling nonlinear robotic systems with unknown dynamics. - [Supporting source](https://arxiv.org/abs/2611.14186) ## Source-stated The authors say linear Koopman dynamics make model-dependent policy-gradient computation a constant matrix product, and one-step prediction limits accumulated prediction errors while the critic retains long-term cost information. > the linear Koopman dynamics simplify the model-dependent policy gradient computation to a constant matrix product - [Supporting source](https://arxiv.org/abs/2611.14186) ## Source-stated The authors report that the learned policy enables real-time control through feedforward evaluation without online optimization. > The learned policy enables real-time control through feedforward evaluation without online optimization. - [Supporting source](https://arxiv.org/abs/2611.14186) ## Source-stated The authors report convergence guarantees under standard stochastic approximation assumptions. > We establish policy learning convergence guarantees under standard stochastic approximation assumptions. - [Supporting source](https://arxiv.org/abs/2611.14186) ## Source-stated The authors report simulation results on HalfCheetah and quadruped locomotion tasks, and validation on a physical quadruped performing a sequence of maneuvers. > The proposed K-MBRL is further validated on a physical quadruped robot performing a challenging sequence of maneuvers - [Supporting source](https://arxiv.org/abs/2611.14186) ## Source-stated The abstract describes the physical robot sequence as rising onto two rounded rear feet, balancing during bipedal walking, and safely returning to the ground. > rising to an upright posture supported by two rounded rear feet, maintaining balance during bipedal walking, and safely returning to the ground - [Supporting source](https://arxiv.org/abs/2611.14186) ## Source and contributor context Hao, Wenjian, Heredia, Paulo C., Mou, Shaoshuai · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Reinforcement learning Training by trial and error, in which a system is rewarded for the outcomes its designers want. - [Definition reference](https://developers.google.com/machine-learning/glossary#reinforcement-learning-rl) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 2b66d2aa-016c-49d2-8c01-078d8a67832f. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T22:33:42.728Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: c0ff1fba6fc27eaede63e18f1f715b511ee5262308bed630a937620f54ac4c61. Source text hash: 30eb1eb5c20f1aabb7bd36aa8e551ccfa1c15288550f8afc434959a846cf8aea. ## Revision and source-check history ### Version 1 — 2026-11-11T22:33:42.728Z Authors Wenjian Hao, Paulo C. Heredia and Shaoshuai Mou propose K-MBRL, a method for controlling nonlinear robotic systems with unknown dynamics. It combines a learned model with a critic, which estimates future cost, to improve a control policy. The authors say representing dynamics with linear Koopman dynamics makes policy-gradient computation a constant matrix product, while using a one-step predicted state avoids repeated long-horizon model predictions. The critic retains information about future cost. The resulting policy acts through feedforward evaluation, without online optimization. The authors report convergence guarantees under standard stochastic approximation assumptions. They say simulations on HalfCheetah and quadruped locomotion tasks show strong sample efficiency and improved performance against classical reinforcement-learning baselines. They also report testing on a physical quadruped: it rose onto two rounded rear feet, balanced during bipedal walking, and returned to the ground. The physical-robot validation extends beyond simulation, but the abstract describes only this sequence of maneuvers. The authors use a pendulum task to assess policy-gradient approximation accuracy and examine components and hyperparameters. ## Original source arXiv eess.SY · Article Original source date: Nov 10, 2026 (2026-11-10). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-11T22:33:42.728Z. First observed: 2026-11-11T05:05:08.537Z. Retrieved: 2026-11-11T22:32:41.713Z. - [Koopman Model-Based Reinforcement Learning](https://arxiv.org/abs/2611.14186) ## Limits The evidence is the arXiv abstract; it reports qualitative comparisons but gives no numerical performance results in the supplied text. The physical quadruped validation described here covers a specific sequence of maneuvers; the supplied abstract does not report performance beyond it. Technical terms to consider for the term library: Koopman model-based reinforcement learning, Koopman operator, critic, policy gradient, feedforward evaluation, and stochastic approximation. ## Open question How does K-MBRL perform on the physical quadruped beyond the described sequence of maneuvers? ## Source-stated The authors propose K-MBRL for controlling nonlinear robotic systems with unknown dynamics, using a learned policy objective that combines immediate cost with estimated future cost. > We propose a Koopman model-based reinforcement learning (K-MBRL) algorithm for controlling nonlinear robotic systems with unknown dynamics. - [Supporting source](https://arxiv.org/abs/2611.14186) ## Source-stated The authors say linear Koopman dynamics make model-dependent policy-gradient computation a constant matrix product, and one-step prediction limits accumulated prediction errors while the critic retains long-term cost information. > the linear Koopman dynamics simplify the model-dependent policy gradient computation to a constant matrix product - [Supporting source](https://arxiv.org/abs/2611.14186) ## Source-stated The authors report that the learned policy enables real-time control through feedforward evaluation without online optimization. > The learned policy enables real-time control through feedforward evaluation without online optimization. - [Supporting source](https://arxiv.org/abs/2611.14186) ## Source-stated The authors report convergence guarantees under standard stochastic approximation assumptions. > We establish policy learning convergence guarantees under standard stochastic approximation assumptions. - [Supporting source](https://arxiv.org/abs/2611.14186) ## Source-stated The authors report simulation results on HalfCheetah and quadruped locomotion tasks, and validation on a physical quadruped performing a sequence of maneuvers. > The proposed K-MBRL is further validated on a physical quadruped robot performing a challenging sequence of maneuvers - [Supporting source](https://arxiv.org/abs/2611.14186) ## Source-stated The abstract describes the physical robot sequence as rising onto two rounded rear feet, balancing during bipedal walking, and safely returning to the ground. > rising to an upright posture supported by two rounded rear feet, maintaining balance during bipedal walking, and safely returning to the ground - [Supporting source](https://arxiv.org/abs/2611.14186) ## Source and contributor context Hao, Wenjian, Heredia, Paulo C., Mou, Shaoshuai · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Reinforcement learning Training by trial and error, in which a system is rewarded for the outcomes its designers want. - [Definition reference](https://developers.google.com/machine-learning/glossary#reinforcement-learning-rl) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 2b66d2aa-016c-49d2-8c01-078d8a67832f. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T22:33:42.728Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: c0ff1fba6fc27eaede63e18f1f715b511ee5262308bed630a937620f54ac4c61. Source text hash: 30eb1eb5c20f1aabb7bd36aa8e551ccfa1c15288550f8afc434959a846cf8aea. ### uncertain — 2026-11-11T22:33:42.728Z — anthropic/claude-opus-5.5 Publication: 2b66d2aa-016c-49d2-8c01-078d8a67832f attribution — pass: The draft correctly credits Wenjian Hao, Paulo C. Heredia and Shaoshuai Mou. It keeps the method, guarantees and results framed as the authors' proposals and reports. > Authors:Wenjian Hao, Paulo C. Heredia, Shaoshuai Mou summary — uncertain: The draft is faithful but drops the payoff of two mechanisms. It says the gradient becomes a constant matrix product but omits the source's point: this enables computationally efficient policy improvement. It recasts the one-step design as avoiding long-horizon predictions, but the source's stated aim is mitigating accumulated prediction errors. It also omits 'safely' from the return to the ground. Central terms (Koopman, policy gradient, feedforward evaluation) rely on the suggested tooltips, and the reader gets no plain sense of why this matters. Smallest repair: add 'enabling computationally efficient policy improvement' and 'to mitigate accumulated prediction errors'. > simplify the model-dependent policy gradient computation to a constant matrix product, enabling computationally efficient policy improvement date — pass: originalPublished 2026-11-10 matches the v1 submission date. The feed date of Nov 11 is later and correctly not used. > \[Submitted on 10 Nov 2026\] support — pass: All claims are stated by the source, with matching excerpts and appropriate hedging. Claim 1 says 'learned policy objective', whereas the source says 'approximate' and the immediate cost is 'analytical'. This is minor imprecision, not a material contradiction. The limitations and the question about performance beyond the described maneuvers follow from the abstract and do not overstate it. > combines the analytical immediate cost with an estimate of the future cost evaluated at the one-step predicted state media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most direct access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/76fb461a8d5d04d317780adf3a768d1e64ab2a89c696a2547f750fab67882213/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-koopman-model-based-reinforcement-learning-76fb461a.json)