Glentis, Athanasios, Li, Dawei, Hong, Mingyi · Oct 27, 2026
MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control
Markdown · JSON · Original source
cs.AI lists a paper by Athanasios Glentis, Dawei Li and Mingyi Hong proposing MuonGS, an optimizer that uses Gram--Schmidt (GS) projection to remove the radial component of each Muon update without fixing the weight norm. The authors analyze accumulated effective learning rate, which measures the sum of the nominal learning rate relative to the weight norm over training. They report that MuonGS guarantees accumulation of at least order square-root in the number of training steps under stated conditions, and establish a stationarity guarantee for smooth scale-invariant objectives. In experiments, MuonGS achieves lower validation perplexity than MuonH at long training horizons. For a dense model, the authors report an estimated token-equivalent speedup of up to about 15% at matched validation loss. The abstract does not establish whether these results extend to other models or training settings.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 27, 2026 (2026-10-27).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-14T01:34:14.753Z. First observed: 2026-10-28T07:54:35.012Z. Retrieved: 2026-10-30T04:16:59.336Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The supplied evidence is the paper’s abstract, not its full methods or results; it does not provide experiment details needed to assess the reported comparisons. The abstract reports results for specific experiments but does not establish whether the gains extend to other models or training settings.
Open question
Do MuonGS’s reported validation and token-equivalent speedup results hold across other model sizes and training settings?
Source-stated
The authors propose MuonGS, which uses Gram--Schmidt projection to remove each Muon update’s radial component without fixing the weight norm.
MuonGS, which uses Gram--Schmidt (GS) projection to remove the radial component of each Muon update without fixing the weight norm.
Source-stated
Under the stated conditions, the authors show MuonGS guarantees accumulated effective learning rate of at least order square-root in the number of training steps.
MuonGS guarantees $C_T=\Omega(\sqrt{T})$, with accumulation sustained even in later training.
Source-stated
For smooth scale-invariant objectives, the authors establish an $O(T^{-1/2})$ stationarity guarantee.
for smooth scale-invariant objectives we establish its $O(T^{-1/2})$ stationarity guarantee.
Source-stated
In the reported pretraining experiments, MuonGS achieved lower validation perplexity than MuonH at long training horizons.
MuonGS achieves lower validation perplexity than MuonH at long training horizons.
Source-stated
For a dense model, the authors report an estimated token-equivalent speedup of up to about 15% at matched validation loss to MuonH.
for the dense model it achieves an estimated token-equivalent speedup up to about $15\%$ at matched validation loss to MuonH.
Source and contributor context
Glentis, Athanasios, Li, Dawei, Hong, Mingyi · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Pre-training
A model's initial training on a large dataset, before further adaptation for particular uses.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 2d2a6a9b-7361-429d-80e9-69648887d1bf.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-14T01:34:14.753Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 1a0428bfc2de251788cf0dd66de1f92e5389eac694f6e9e5d43bb50af68a576b. Source text hash: ad07f1b580396795f9691600dbeb9fb99b569550d71bc92b6707ea07616f9248.
Revision and source-check history
Version 1 · 2026-11-14T01:34:14.753Z
MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control
cs.AI lists a paper by Athanasios Glentis, Dawei Li and Mingyi Hong proposing MuonGS, an optimizer that uses Gram--Schmidt (GS) projection to remove the radial component of each Muon update without fixing the weight norm. The authors analyze accumulated effective learning rate, which measures the sum of the nominal learning rate relative to the weight norm over training. They report that MuonGS guarantees accumulation of at least order square-root in the number of training steps under stated conditions, and establish a stationarity guarantee for smooth scale-invariant objectives. In experiments, MuonGS achieves lower validation perplexity than MuonH at long training horizons. For a dense model, the authors report an estimated token-equivalent speedup of up to about 15% at matched validation loss. The abstract does not establish whether these results extend to other models or training settings.
Original source
cs.AI · Article
Source details
Original source date: Oct 27, 2026 (2026-10-27).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-14T01:34:14.753Z. First observed: 2026-10-28T07:54:35.012Z. Retrieved: 2026-10-30T04:16:59.336Z.
Limits
The supplied evidence is the paper’s abstract, not its full methods or results; it does not provide experiment details needed to assess the reported comparisons. The abstract reports results for specific experiments but does not establish whether the gains extend to other models or training settings.
Open question
Do MuonGS’s reported validation and token-equivalent speedup results hold across other model sizes and training settings?
Source-stated
The authors propose MuonGS, which uses Gram--Schmidt projection to remove each Muon update’s radial component without fixing the weight norm.
MuonGS, which uses Gram--Schmidt (GS) projection to remove the radial component of each Muon update without fixing the weight norm.
Source-stated
Under the stated conditions, the authors show MuonGS guarantees accumulated effective learning rate of at least order square-root in the number of training steps.
MuonGS guarantees $C_T=\Omega(\sqrt{T})$, with accumulation sustained even in later training.
Source-stated
For smooth scale-invariant objectives, the authors establish an $O(T^{-1/2})$ stationarity guarantee.
for smooth scale-invariant objectives we establish its $O(T^{-1/2})$ stationarity guarantee.
Source-stated
In the reported pretraining experiments, MuonGS achieved lower validation perplexity than MuonH at long training horizons.
MuonGS achieves lower validation perplexity than MuonH at long training horizons.
Source-stated
For a dense model, the authors report an estimated token-equivalent speedup of up to about 15% at matched validation loss to MuonH.
for the dense model it achieves an estimated token-equivalent speedup up to about $15\%$ at matched validation loss to MuonH.
Source and contributor context
Glentis, Athanasios, Li, Dawei, Hong, Mingyi · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Pre-training
A model's initial training on a large dataset, before further adaptation for particular uses.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 2d2a6a9b-7361-429d-80e9-69648887d1bf.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-14T01:34:14.753Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 1a0428bfc2de251788cf0dd66de1f92e5389eac694f6e9e5d43bb50af68a576b. Source text hash: ad07f1b580396795f9691600dbeb9fb99b569550d71bc92b6707ea07616f9248.
uncertain · 2026-11-14T01:34:14.753Z · anthropic/claude-opus-5.5
Publication: 2d2a6a9b-7361-429d-80e9-69648887d1bf
attribution — pass: The paper is correctly attributed to Athanasios Glentis, Dawei Li and Mingyi Hong. Findings are framed as the authors' claims. arXiv lists the primary category as cs.LG with a cs.AI cross-list, so the cs.AI listing is acceptable.
Authors:Athanasios Glentis, Dawei Li, Mingyi Hong
summary — uncertain: Central terms are left unexplained for a general reader: optimizer, Muon, MuonH, weight norm, radial component, validation perplexity, stationarity and scale-invariant objectives. The reading also drops the paper's motivating argument: vanilla Muon accumulates only logarithmically, which may limit exploration and help explain its slower progress, while slow norm growth allows faster accumulation. Without this, readers cannot see why MuonGS matters. Smallest repair: add one sentence on that contrast and plain-language tooltips for Muon, MuonH and perplexity. Also note that tests used 160M and 400M dense models and a MoE model.
We show that vanilla Muon can accumulate only logarithmically, potentially limiting exploration and helping explain its slower empirical progress
date — pass: originalPublished 2026-10-27 matches the v1 submission date, not the later feed date.
[Submitted on 27 Oct 2026]
support — pass: Each claim matches the abstract and is qualified as stated conditions or reported results. Wording such as 'estimated', 'up to about 15%' and 'at long training horizons' is preserved. The limitations and question do not overstate generalization or real-world benefit.
MuonGS achieves lower validation perplexity than MuonH at long training horizons
media — pass: The selected media is the verified direct link to the original arXiv abstract page, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA model's initial training on a large dataset, before further adaptation for particular uses.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA model's initial training on a large dataset, before further adaptation for particular uses.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA model's initial training on a large dataset, before further adaptation for particular uses.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA model's initial training on a large dataset, before further adaptation for particular uses.
Definition reference