Xie, Zirong, Afolabi, Zainab, Kozodoi, Nikita, Butler, Jack · Nov 4, 2026
arXiv cs.AI: Model Merging Does Not Require Uniform LoRA Ranks
Markdown · JSON · Original source
In an cs.AI abstract, the authors of “Model Merging Does Not Require Uniform LoRA Ranks” argue that model merging need not force separately experts to use the same LoRA rank. They report experiments with 36 experts across mathematics, instruction following and multilingual tasks, using four ranks, three model scales and six merging methods. The authors say averaging and utility-based selection gained up to 1.8 percentage points in multi-domain accuracy when each domain used its own best rank, while conflict-resolution methods were inconsistent and lost up to 4 points. They attribute the difference to how merging methods select spectral components: methods selecting by size can discard components that would reduce loss and nearly drop the instruction adapter, which they link to a 16-to-25-point fall on IFEval. By contrast, selecting components by their effect on loss kept all three experts and left accuracy nearly flat across an 8× rank range in the reported comparison. The authors conclude that rank can be chosen separately for each expert when the merge operator does not select by magnitude. Their results make the choice of merging method—not a shared rank—the key condition in their argument; how broadly that condition holds beyond the reported experiments remains open.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 4, 2026 (2026-11-04).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T08:23:33.799Z. First observed: 2026-11-05T05:06:22.287Z. Retrieved: 2026-11-09T08:17:09.331Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
This reading is based on the abstract, which reports the experimental setup and headline results but does not provide the detailed methods or results needed to assess them independently. The reported experiments cover 36 experts, three domains, four ranks, three model scales and six merging methods; the abstract does not establish how the results generalize beyond those tested combinations.
Open question
How do these results vary across domains, model scales and merging methods beyond the combinations tested in the paper?
Source-stated
The authors argue that requiring every expert to use the same LoRA rank is a property of the merge method, not a requirement of model merging.
We show this coupling is a property of the merge method, not of merging.
Source-stated
The authors report that allowing domain-specific ranks cost averaging and utility-based selection nothing and improved multi-domain accuracy by up to 1.8 percentage points; conflict-resolution methods were inconsistent and lost up to 4 points.
Giving experts different ranks costs averaging and utility-based selection nothing, and gains them up to 1.8 pp of multi-domain accuracy, while the conflict-resolution methods are inconsistent and can lose up to 4 pp.
Source-stated
The authors attribute the difference to selection by size discarding components that would have reduced loss; they report that the instruction adapter was nearly discarded and IFEval fell by 16 to 25 percentage points.
methods that choose them by size discard many that would have reduced loss. They also tend to discard the same expert: the instruction adapter has the smallest singular values, so it is dropped almost entirely and IFEval falls by 16 to 25 pp.
Source-stated
The authors say selecting components by their effect on loss kept all three experts and left accuracy nearly flat across an 8× rank range, compared with a 6.7-point change for TIES at 4B scale.
Choosing components by their effect on loss instead keeps all three experts, and leaves accuracy nearly flat across an 8$\times$ rank range (0.7 pp, against 6.7 pp for TIES at 4B scale).
Source-stated
The authors conclude that rank can be chosen separately for each expert when the merge operator does not select by magnitude.
Rank can therefore be chosen per expert whenever the merge operator does not select by magnitude.
Source and contributor context
Xie, Zirong, Afolabi, Zainab, Kozodoi, Nikita, Butler, Jack · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Fine-tuning
Further training of an existing model to adapt it to a particular task or use.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 51673607-eb97-4086-828a-91206e73ba10.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T08:23:33.799Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 690ab8bd07cef4fd072ba6682b0ab980c9a854fe82f40987bc9d4f9de8513737. Source text hash: 53b93303ce85b40a4e8536e5f106ad68c0e1dcb53cd6ecc29c03c0667831d64d.
Revision and source-check history
Version 1 · 2026-11-09T08:23:33.799Z
arXiv cs.AI: Model Merging Does Not Require Uniform LoRA Ranks
In an cs.AI abstract, the authors of “Model Merging Does Not Require Uniform LoRA Ranks” argue that model merging need not force separately experts to use the same LoRA rank. They report experiments with 36 experts across mathematics, instruction following and multilingual tasks, using four ranks, three model scales and six merging methods. The authors say averaging and utility-based selection gained up to 1.8 percentage points in multi-domain accuracy when each domain used its own best rank, while conflict-resolution methods were inconsistent and lost up to 4 points. They attribute the difference to how merging methods select spectral components: methods selecting by size can discard components that would reduce loss and nearly drop the instruction adapter, which they link to a 16-to-25-point fall on IFEval. By contrast, selecting components by their effect on loss kept all three experts and left accuracy nearly flat across an 8× rank range in the reported comparison. The authors conclude that rank can be chosen separately for each expert when the merge operator does not select by magnitude. Their results make the choice of merging method—not a shared rank—the key condition in their argument; how broadly that condition holds beyond the reported experiments remains open.
Original source
cs.AI · Article
Source details
Original source date: Nov 4, 2026 (2026-11-04).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T08:23:33.799Z. First observed: 2026-11-05T05:06:22.287Z. Retrieved: 2026-11-09T08:17:09.331Z.
Limits
This reading is based on the abstract, which reports the experimental setup and headline results but does not provide the detailed methods or results needed to assess them independently. The reported experiments cover 36 experts, three domains, four ranks, three model scales and six merging methods; the abstract does not establish how the results generalize beyond those tested combinations.
Open question
How do these results vary across domains, model scales and merging methods beyond the combinations tested in the paper?
Source-stated
The authors argue that requiring every expert to use the same LoRA rank is a property of the merge method, not a requirement of model merging.
We show this coupling is a property of the merge method, not of merging.
Source-stated
The authors report that allowing domain-specific ranks cost averaging and utility-based selection nothing and improved multi-domain accuracy by up to 1.8 percentage points; conflict-resolution methods were inconsistent and lost up to 4 points.
Giving experts different ranks costs averaging and utility-based selection nothing, and gains them up to 1.8 pp of multi-domain accuracy, while the conflict-resolution methods are inconsistent and can lose up to 4 pp.
Source-stated
The authors attribute the difference to selection by size discarding components that would have reduced loss; they report that the instruction adapter was nearly discarded and IFEval fell by 16 to 25 percentage points.
methods that choose them by size discard many that would have reduced loss. They also tend to discard the same expert: the instruction adapter has the smallest singular values, so it is dropped almost entirely and IFEval falls by 16 to 25 pp.
Source-stated
The authors say selecting components by their effect on loss kept all three experts and left accuracy nearly flat across an 8× rank range, compared with a 6.7-point change for TIES at 4B scale.
Choosing components by their effect on loss instead keeps all three experts, and leaves accuracy nearly flat across an 8$\times$ rank range (0.7 pp, against 6.7 pp for TIES at 4B scale).
Source-stated
The authors conclude that rank can be chosen separately for each expert when the merge operator does not select by magnitude.
Rank can therefore be chosen per expert whenever the merge operator does not select by magnitude.
Source and contributor context
Xie, Zirong, Afolabi, Zainab, Kozodoi, Nikita, Butler, Jack · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Fine-tuning
Further training of an existing model to adapt it to a particular task or use.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 51673607-eb97-4086-828a-91206e73ba10.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T08:23:33.799Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 690ab8bd07cef4fd072ba6682b0ab980c9a854fe82f40987bc9d4f9de8513737. Source text hash: 53b93303ce85b40a4e8536e5f106ad68c0e1dcb53cd6ecc29c03c0667831d64d.
uncertain · 2026-11-09T08:23:33.799Z · anthropic/claude-opus-5.5
Publication: 51673607-eb97-4086-828a-91206e73ba10
attribution — pass: The reading consistently attributes the claims to the paper's authors and presents them as reported findings rather than established facts. The arXiv listing is primarily cs.LG, cross-listed to cs.AI. The 'arXiv cs.AI' label matches the feed publisher and does not misattribute the work.
Authors:Zirong Xie, Zainab Afolabi, Nikita Kozodoi, Jack Butler
summary — uncertain: The figures and conditions are faithful to the abstract. However, central terms are not explained for a non-specialist reader: LoRA rank, spectral components, singular values or magnitude selection, IFEval, TIES and merge operator. The draft also omits the abstract's plain definition that merging combines fine-tuned experts without retraining. A reader cannot follow why rank or selection by size matters. Smallest repair: add the abstract's definition of merging, and add term-library tooltips for LoRA rank, IFEval and TIES, without paraphrasing those terms.
Model merging combines separately fine-tuned experts into a single model without retraining.
date — pass: The originalPublished date of 2026-11-04 matches the v1 submission date. It is not the later feed entry date.
[Submitted on 4 Nov 2026]
support — pass: Every claim's excerpt is verbatim and supports its claim, including 1.8 pp, up to 4 pp, 16 to 25 pp on IFEval, and 0.7 pp against 6.7 pp for TIES at 4B scale. The conditional conclusion is preserved. The closing sentence and the question are appropriately framed as open rather than asserting generalization.
Rank can therefore be chosen per expert whenever the merge operator does not select by magnitude.
media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivFurther training of an existing model to adapt it to a particular task or use.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivFurther training of an existing model to adapt it to a particular task or use.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv