Hao, Wenjian, Heredia, Paulo C., Mou, Shaoshuai · Nov 10, 2026
arXiv: Koopman Model-Based Reinforcement Learning
Markdown · JSON · Original source
Authors Wenjian Hao, Paulo C. Heredia and Shaoshuai Mou propose K-MBRL, a method for controlling nonlinear robotic systems with unknown dynamics. It combines a learned model with a critic, which estimates future cost, to improve a control policy. The authors say representing dynamics with linear Koopman dynamics makes policy-gradient computation a constant matrix product, while using a one-step predicted state avoids repeated long-horizon model predictions. The critic retains information about future cost. The resulting policy acts through feedforward evaluation, without online optimization. The authors report convergence guarantees under standard stochastic approximation assumptions. They say simulations on HalfCheetah and quadruped locomotion tasks show strong sample efficiency and improved performance against classical baselines. They also report testing on a physical quadruped: it rose onto two rounded rear feet, balanced during bipedal walking, and returned to the ground. The physical-robot validation extends beyond simulation, but the abstract describes only this sequence of maneuvers. The authors use a pendulum task to assess policy-gradient approximation accuracy and examine components and hyperparameters.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
eess.SY · Article
Source details
Original source date: Nov 10, 2026 (2026-11-10).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-11T22:33:42.728Z. First observed: 2026-11-11T05:05:08.537Z. Retrieved: 2026-11-11T22:32:41.713Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The evidence is the abstract; it reports qualitative comparisons but gives no numerical performance results in the supplied text. The physical quadruped validation described here covers a specific sequence of maneuvers; the supplied abstract does not report performance beyond it. Technical terms to consider for the term library: Koopman model-based , Koopman operator, critic, policy gradient, feedforward evaluation, and stochastic approximation.
Open question
How does K-MBRL perform on the physical quadruped beyond the described sequence of maneuvers?
Source-stated
The authors propose K-MBRL for controlling nonlinear robotic systems with unknown dynamics, using a learned policy objective that combines immediate cost with estimated future cost.
We propose a Koopman model-based reinforcement learning (K-MBRL) algorithm for controlling nonlinear robotic systems with unknown dynamics.
Source-stated
The authors say linear Koopman dynamics make model-dependent policy-gradient computation a constant matrix product, and one-step prediction limits accumulated prediction errors while the critic retains long-term cost information.
the linear Koopman dynamics simplify the model-dependent policy gradient computation to a constant matrix product
Source-stated
The authors report that the learned policy enables real-time control through feedforward evaluation without online optimization.
The learned policy enables real-time control through feedforward evaluation without online optimization.
Source-stated
The authors report convergence guarantees under standard stochastic approximation assumptions.
We establish policy learning convergence guarantees under standard stochastic approximation assumptions.
Source-stated
The authors report simulation results on HalfCheetah and quadruped locomotion tasks, and validation on a physical quadruped performing a sequence of maneuvers.
The proposed K-MBRL is further validated on a physical quadruped robot performing a challenging sequence of maneuvers
Source-stated
The abstract describes the physical robot sequence as rising onto two rounded rear feet, balancing during bipedal walking, and safely returning to the ground.
rising to an upright posture supported by two rounded rear feet, maintaining balance during bipedal walking, and safely returning to the ground
Source and contributor context
Hao, Wenjian, Heredia, Paulo C., Mou, Shaoshuai · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Reinforcement learning
Training by trial and error, in which a system is rewarded for the outcomes its designers want.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 2b66d2aa-016c-49d2-8c01-078d8a67832f.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T22:33:42.728Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: c0ff1fba6fc27eaede63e18f1f715b511ee5262308bed630a937620f54ac4c61. Source text hash: 30eb1eb5c20f1aabb7bd36aa8e551ccfa1c15288550f8afc434959a846cf8aea.
Revision and source-check history
Version 1 · 2026-11-11T22:33:42.728Z
arXiv: Koopman Model-Based Reinforcement Learning
Authors Wenjian Hao, Paulo C. Heredia and Shaoshuai Mou propose K-MBRL, a method for controlling nonlinear robotic systems with unknown dynamics. It combines a learned model with a critic, which estimates future cost, to improve a control policy. The authors say representing dynamics with linear Koopman dynamics makes policy-gradient computation a constant matrix product, while using a one-step predicted state avoids repeated long-horizon model predictions. The critic retains information about future cost. The resulting policy acts through feedforward evaluation, without online optimization. The authors report convergence guarantees under standard stochastic approximation assumptions. They say simulations on HalfCheetah and quadruped locomotion tasks show strong sample efficiency and improved performance against classical baselines. They also report testing on a physical quadruped: it rose onto two rounded rear feet, balanced during bipedal walking, and returned to the ground. The physical-robot validation extends beyond simulation, but the abstract describes only this sequence of maneuvers. The authors use a pendulum task to assess policy-gradient approximation accuracy and examine components and hyperparameters.
Original source
eess.SY · Article
Source details
Original source date: Nov 10, 2026 (2026-11-10).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-11T22:33:42.728Z. First observed: 2026-11-11T05:05:08.537Z. Retrieved: 2026-11-11T22:32:41.713Z.
Limits
The evidence is the abstract; it reports qualitative comparisons but gives no numerical performance results in the supplied text. The physical quadruped validation described here covers a specific sequence of maneuvers; the supplied abstract does not report performance beyond it. Technical terms to consider for the term library: Koopman model-based , Koopman operator, critic, policy gradient, feedforward evaluation, and stochastic approximation.
Open question
How does K-MBRL perform on the physical quadruped beyond the described sequence of maneuvers?
Source-stated
The authors propose K-MBRL for controlling nonlinear robotic systems with unknown dynamics, using a learned policy objective that combines immediate cost with estimated future cost.
We propose a Koopman model-based reinforcement learning (K-MBRL) algorithm for controlling nonlinear robotic systems with unknown dynamics.
Source-stated
The authors say linear Koopman dynamics make model-dependent policy-gradient computation a constant matrix product, and one-step prediction limits accumulated prediction errors while the critic retains long-term cost information.
the linear Koopman dynamics simplify the model-dependent policy gradient computation to a constant matrix product
Source-stated
The authors report that the learned policy enables real-time control through feedforward evaluation without online optimization.
The learned policy enables real-time control through feedforward evaluation without online optimization.
Source-stated
The authors report convergence guarantees under standard stochastic approximation assumptions.
We establish policy learning convergence guarantees under standard stochastic approximation assumptions.
Source-stated
The authors report simulation results on HalfCheetah and quadruped locomotion tasks, and validation on a physical quadruped performing a sequence of maneuvers.
The proposed K-MBRL is further validated on a physical quadruped robot performing a challenging sequence of maneuvers
Source-stated
The abstract describes the physical robot sequence as rising onto two rounded rear feet, balancing during bipedal walking, and safely returning to the ground.
rising to an upright posture supported by two rounded rear feet, maintaining balance during bipedal walking, and safely returning to the ground
Source and contributor context
Hao, Wenjian, Heredia, Paulo C., Mou, Shaoshuai · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Reinforcement learning
Training by trial and error, in which a system is rewarded for the outcomes its designers want.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 2b66d2aa-016c-49d2-8c01-078d8a67832f.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T22:33:42.728Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: c0ff1fba6fc27eaede63e18f1f715b511ee5262308bed630a937620f54ac4c61. Source text hash: 30eb1eb5c20f1aabb7bd36aa8e551ccfa1c15288550f8afc434959a846cf8aea.
uncertain · 2026-11-11T22:33:42.728Z · anthropic/claude-opus-5.5
Publication: 2b66d2aa-016c-49d2-8c01-078d8a67832f
attribution — pass: The draft correctly credits Wenjian Hao, Paulo C. Heredia and Shaoshuai Mou. It keeps the method, guarantees and results framed as the authors' proposals and reports.
Authors:Wenjian Hao, Paulo C. Heredia, Shaoshuai Mou
summary — uncertain: The draft is faithful but drops the payoff of two mechanisms. It says the gradient becomes a constant matrix product but omits the source's point: this enables computationally efficient policy improvement. It recasts the one-step design as avoiding long-horizon predictions, but the source's stated aim is mitigating accumulated prediction errors. It also omits 'safely' from the return to the ground. Central terms (Koopman, policy gradient, feedforward evaluation) rely on the suggested tooltips, and the reader gets no plain sense of why this matters. Smallest repair: add 'enabling computationally efficient policy improvement' and 'to mitigate accumulated prediction errors'.
simplify the model-dependent policy gradient computation to a constant matrix product, enabling computationally efficient policy improvement
date — pass: originalPublished 2026-11-10 matches the v1 submission date. The feed date of Nov 11 is later and correctly not used.
[Submitted on 10 Nov 2026]
support — pass: All claims are stated by the source, with matching excerpts and appropriate hedging. Claim 1 says 'learned policy objective', whereas the source says 'approximate' and the immediate cost is 'analytical'. This is minor imprecision, not a material contradiction. The limitations and the question about performance beyond the described maneuvers follow from the abstract and do not overstate it.
combines the analytical immediate cost with an estimate of the future cost evaluated at the one-step predicted state
media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Training by trial and error, in which a system is rewarded for the outcomes its designers want.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivTraining by trial and error, in which a system is rewarded for the outcomes its designers want.
Definition referenceTraining by trial and error, in which a system is rewarded for the outcomes its designers want.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivTraining by trial and error, in which a system is rewarded for the outcomes its designers want.
Definition reference