# MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control [Canonical reading](https://escapemeridian.com/readings/muongs-pretraining-llms-with-relaxed-weight-norm-control-977870e4) arXiv cs.AI lists a paper by Athanasios Glentis, Dawei Li and Mingyi Hong proposing MuonGS, an optimizer that uses Gram--Schmidt (GS) projection to remove the radial component of each Muon update without fixing the weight norm. The authors analyze accumulated effective learning rate, which measures the sum of the nominal learning rate relative to the weight norm over training. They report that MuonGS guarantees accumulation of at least order square-root in the number of training steps under stated conditions, and establish a stationarity guarantee for smooth scale-invariant objectives. In pretraining experiments, MuonGS achieves lower validation perplexity than MuonH at long training horizons. For a dense model, the authors report an estimated token-equivalent speedup of up to about 15% at matched validation loss. The abstract does not establish whether these results extend to other models or training settings. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Oct 27, 2026 (2026-10-27). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-14T01:34:14.753Z. First observed: 2026-10-28T07:54:35.012Z. Retrieved: 2026-10-30T04:16:59.336Z. - [MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control](https://arxiv.org/abs/2610.37647) ## Limits The supplied evidence is the paper’s abstract, not its full methods or results; it does not provide experiment details needed to assess the reported comparisons. The abstract reports results for specific pretraining experiments but does not establish whether the gains extend to other models or training settings. ## Open question Do MuonGS’s reported validation and token-equivalent speedup results hold across other model sizes and training settings? ## Source-stated The authors propose MuonGS, which uses Gram--Schmidt projection to remove each Muon update’s radial component without fixing the weight norm. > MuonGS, which uses Gram--Schmidt (GS) projection to remove the radial component of each Muon update without fixing the weight norm. - [Supporting source](https://arxiv.org/abs/2610.37647) ## Source-stated Under the stated conditions, the authors show MuonGS guarantees accumulated effective learning rate of at least order square-root in the number of training steps. > MuonGS guarantees $C\_T=\\Omega(\\sqrt{T})$, with accumulation sustained even in later training. - [Supporting source](https://arxiv.org/abs/2610.37647) ## Source-stated For smooth scale-invariant objectives, the authors establish an $O(T^{-1/2})$ stationarity guarantee. > for smooth scale-invariant objectives we establish its $O(T^{-1/2})$ stationarity guarantee. - [Supporting source](https://arxiv.org/abs/2610.37647) ## Source-stated In the reported pretraining experiments, MuonGS achieved lower validation perplexity than MuonH at long training horizons. > MuonGS achieves lower validation perplexity than MuonH at long training horizons. - [Supporting source](https://arxiv.org/abs/2610.37647) ## Source-stated For a dense model, the authors report an estimated token-equivalent speedup of up to about 15% at matched validation loss to MuonH. > for the dense model it achieves an estimated token-equivalent speedup up to about $15\\%$ at matched validation loss to MuonH. - [Supporting source](https://arxiv.org/abs/2610.37647) ## Source and contributor context Glentis, Athanasios, Li, Dawei, Hong, Mingyi · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Pre-training A model's initial training on a large dataset, before further adaptation for particular uses. - [Definition reference](https://developers.google.com/machine-learning/glossary#pre-training) ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 2d2a6a9b-7361-429d-80e9-69648887d1bf. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-14T01:34:14.753Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 1a0428bfc2de251788cf0dd66de1f92e5389eac694f6e9e5d43bb50af68a576b. Source text hash: ad07f1b580396795f9691600dbeb9fb99b569550d71bc92b6707ea07616f9248. ## Revision and source-check history ### Version 1 — 2026-11-14T01:34:14.753Z arXiv cs.AI lists a paper by Athanasios Glentis, Dawei Li and Mingyi Hong proposing MuonGS, an optimizer that uses Gram--Schmidt (GS) projection to remove the radial component of each Muon update without fixing the weight norm. The authors analyze accumulated effective learning rate, which measures the sum of the nominal learning rate relative to the weight norm over training. They report that MuonGS guarantees accumulation of at least order square-root in the number of training steps under stated conditions, and establish a stationarity guarantee for smooth scale-invariant objectives. In pretraining experiments, MuonGS achieves lower validation perplexity than MuonH at long training horizons. For a dense model, the authors report an estimated token-equivalent speedup of up to about 15% at matched validation loss. The abstract does not establish whether these results extend to other models or training settings. ## Original source arXiv cs.AI · Article Original source date: Oct 27, 2026 (2026-10-27). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-14T01:34:14.753Z. First observed: 2026-10-28T07:54:35.012Z. Retrieved: 2026-10-30T04:16:59.336Z. - [MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control](https://arxiv.org/abs/2610.37647) ## Limits The supplied evidence is the paper’s abstract, not its full methods or results; it does not provide experiment details needed to assess the reported comparisons. The abstract reports results for specific pretraining experiments but does not establish whether the gains extend to other models or training settings. ## Open question Do MuonGS’s reported validation and token-equivalent speedup results hold across other model sizes and training settings? ## Source-stated The authors propose MuonGS, which uses Gram--Schmidt projection to remove each Muon update’s radial component without fixing the weight norm. > MuonGS, which uses Gram--Schmidt (GS) projection to remove the radial component of each Muon update without fixing the weight norm. - [Supporting source](https://arxiv.org/abs/2610.37647) ## Source-stated Under the stated conditions, the authors show MuonGS guarantees accumulated effective learning rate of at least order square-root in the number of training steps. > MuonGS guarantees $C\_T=\\Omega(\\sqrt{T})$, with accumulation sustained even in later training. - [Supporting source](https://arxiv.org/abs/2610.37647) ## Source-stated For smooth scale-invariant objectives, the authors establish an $O(T^{-1/2})$ stationarity guarantee. > for smooth scale-invariant objectives we establish its $O(T^{-1/2})$ stationarity guarantee. - [Supporting source](https://arxiv.org/abs/2610.37647) ## Source-stated In the reported pretraining experiments, MuonGS achieved lower validation perplexity than MuonH at long training horizons. > MuonGS achieves lower validation perplexity than MuonH at long training horizons. - [Supporting source](https://arxiv.org/abs/2610.37647) ## Source-stated For a dense model, the authors report an estimated token-equivalent speedup of up to about 15% at matched validation loss to MuonH. > for the dense model it achieves an estimated token-equivalent speedup up to about $15\\%$ at matched validation loss to MuonH. - [Supporting source](https://arxiv.org/abs/2610.37647) ## Source and contributor context Glentis, Athanasios, Li, Dawei, Hong, Mingyi · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Pre-training A model's initial training on a large dataset, before further adaptation for particular uses. - [Definition reference](https://developers.google.com/machine-learning/glossary#pre-training) ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 2d2a6a9b-7361-429d-80e9-69648887d1bf. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-14T01:34:14.753Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 1a0428bfc2de251788cf0dd66de1f92e5389eac694f6e9e5d43bb50af68a576b. Source text hash: ad07f1b580396795f9691600dbeb9fb99b569550d71bc92b6707ea07616f9248. ### uncertain — 2026-11-14T01:34:14.753Z — anthropic/claude-opus-5.5 Publication: 2d2a6a9b-7361-429d-80e9-69648887d1bf attribution — pass: The paper is correctly attributed to Athanasios Glentis, Dawei Li and Mingyi Hong. Findings are framed as the authors' claims. arXiv lists the primary category as cs.LG with a cs.AI cross-list, so the cs.AI listing is acceptable. > Authors:Athanasios Glentis, Dawei Li, Mingyi Hong summary — uncertain: Central terms are left unexplained for a general reader: optimizer, Muon, MuonH, weight norm, radial component, validation perplexity, stationarity and scale-invariant objectives. The reading also drops the paper's motivating argument: vanilla Muon accumulates only logarithmically, which may limit exploration and help explain its slower progress, while slow norm growth allows faster accumulation. Without this, readers cannot see why MuonGS matters. Smallest repair: add one sentence on that contrast and plain-language tooltips for Muon, MuonH and perplexity. Also note that tests used 160M and 400M dense models and a MoE model. > We show that vanilla Muon can accumulate only logarithmically, potentially limiting exploration and helping explain its slower empirical progress date — pass: originalPublished 2026-10-27 matches the v1 submission date, not the later feed date. > \[Submitted on 27 Oct 2026\] support — pass: Each claim matches the abstract and is qualified as stated conditions or reported results. Wording such as 'estimated', 'up to about 15%' and 'at long training horizons' is preserved. The limitations and question do not overstate generalization or real-world benefit. > MuonGS achieves lower validation perplexity than MuonH at long training horizons media — pass: The selected media is the verified direct link to the original arXiv abstract page, which is the most direct access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/977870e452262b0b696a9a5ac2bd633e0b6f5da4972ebe2899041979af8ae6b8/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/muongs-pretraining-llms-with-relaxed-weight-norm-control-977870e4.json)