# Your Experts Have Partners. There's HOPE for Keeping Them Together [Canonical reading](https://escapemeridian.com/readings/your-experts-have-partners-there-s-hope-for-keeping-them-ab90bf4c) Amazon author Alex M. Tseng presents HOPE, a method for pruning experts from Mixture-of-Experts (MoE) language models. Unlike first-order methods, which score experts independently, HOPE accounts for pairwise interactions: experts that often activate together may form cooperative groups, so removing one can impair the contribution of another. The authors say this matters most when pruning aggressively, because more cooperative links may be broken. The post reports evaluations on three models, two calibration datasets and several benchmarks. Across the tested conditions, HOPE achieved the best average rank among five methods; at 50% pruning, its average rank was 1.58 versus 2.42 for the next-best method. On Qwen3.5-35B-A3B at that pruning rate, the authors report a score of 42.3 on SWE-Bench Pro, compared with 41.8 for the unpruned model and 36.2 for REAP. The authors describe the gains as most pronounced at 40–50% pruning, while first-order methods are sometimes competitive at 10–20%. HOPE accounts for pairwise interactions within each layer, not cross-layer interactions; the authors say their cross-layer results remain preliminary. How would HOPE’s results change if it accounted for cross-layer interactions or interactions among three or more experts? Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source Hugging Face · Article Original source date: Nov 9, 2026 (2026-11-09T12:34:29.000Z). Date basis: Date supplied by the publisher’s feed. Added to Escape Meridian: 2026-11-10T17:03:59.661Z. First observed: 2026-11-09T12:36:33.153Z. Retrieved: 2026-11-09T12:45:40.946Z. - [Your Experts Have Partners. There's HOPE for Keeping Them Together](https://huggingface.co/blog/amazon/higher-order-pruning-experts) ## Limits The reported results cover three models, two calibration datasets and the benchmarks described in the post; they do not establish performance beyond those evaluations. The authors say HOPE's advantage shrinks at lower pruning rates, where first-order methods are sometimes competitive. HOPE does not account for cross-layer interactions, and the authors describe their cross-layer results as preliminary. HOPE models pairwise interactions; the post says interactions among three or more experts could provide additional signal but are difficult to estimate. ## Open question How would HOPE’s results change if it accounted for cross-layer interactions or interactions among three or more experts? ## Source-stated Alex M. Tseng presents HOPE, a pruning method that accounts for pairwise expert interactions in Mixture-of-Experts models. > Our newest paper derives HOPE (Higher-Order Pruning of Experts), a second-order method that accounts for pairwise expert interactions. - [Supporting source](https://huggingface.co/blog/amazon/higher-order-pruning-experts) ## Source-stated The authors report HOPE achieved the best average rank across the tested models, calibration sets and benchmarks. > Across all conditions, HOPE achieved the best average rank (2.07 out of 5 methods) - [Supporting source](https://huggingface.co/blog/amazon/higher-order-pruning-experts) ## Source-stated At 50% pruning, the authors report HOPE's average rank was 1.58, compared with 2.42 for REAP. > At 50% pruning, HOPE's average rank improves to 1.58 (vs. 2.42 for the next-best method, REAP). - [Supporting source](https://huggingface.co/blog/amazon/higher-order-pruning-experts) ## Source-stated On Qwen3.5-35B-A3B at 50% pruning, the authors report HOPE scored 42.3 on SWE-Bench Pro, versus 41.8 for the unpruned model and 36.2 for REAP. > On Qwen3.5-35B-A3B at 50% pruning, HOPE achieved 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8 (while REAP dropped to 36.2, a 13% regression). - [Supporting source](https://huggingface.co/blog/amazon/higher-order-pruning-experts) ## Source-stated The authors say HOPE's advantage is most pronounced at 40–50% pruning; at 10–20%, first-order methods are sometimes competitive. > HOPE's advantage is most pronounced at higher pruning rates (40–50%). At lower pruning rates (10–20%), first-order methods like REAP and MAN are sometimes competitive - [Supporting source](https://huggingface.co/blog/amazon/higher-order-pruning-experts) ## Source-stated HOPE prunes layers independently and does not account for cross-layer interactions; the authors describe their cross-layer results as preliminary. > HOPE prunes each layer independently and does not account for cross-layer interactions. In our paper, we presented a theoretical framework for cross-layer HOPE, but the results remain preliminary. - [Supporting source](https://huggingface.co/blog/amazon/higher-order-pruning-experts) ## Source-stated HOPE captures pairwise interactions, while interactions among three or more experts could provide additional signal, according to the authors. > Finally, HOPE captures pairwise interactions, but even higher-order interactions (triplets and beyond) could provide additional signal - [Supporting source](https://huggingface.co/blog/amazon/higher-order-pruning-experts) ## Source and contributor context Hugging Face · Publisher The publisher is identified here. No individual author has been inferred. ## Source media Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Mixture of experts A model design that sends each input to a few specialized sub-networks instead of the whole model, which saves computation. - [Definition reference](https://developers.google.com/machine-learning/glossary#mixture-of-experts) ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Hugging Face Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems. - [Explore this context](/entities/hugging-face) - [Reference](https://huggingface.co/blog) ## Editorial provenance Version 1. Publication ID: 744e4ab7-f45d-498a-8a6b-9461b7e96541. Prepared by gpt-6-luna; stream-draft-repair-v1; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T17:03:59.661Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: b65c78caee9e22b49fc9227c418eaf1caca12e369ca29818a159c67d4222911f. Source text hash: 74d15c233a49ec339d9b1eee5a02633ec8c8406e33d7c881b3532315f4c67f61. ## Revision and source-check history ### Version 1 — 2026-11-10T17:03:59.661Z Amazon author Alex M. Tseng presents HOPE, a method for pruning experts from Mixture-of-Experts (MoE) language models. Unlike first-order methods, which score experts independently, HOPE accounts for pairwise interactions: experts that often activate together may form cooperative groups, so removing one can impair the contribution of another. The authors say this matters most when pruning aggressively, because more cooperative links may be broken. The post reports evaluations on three models, two calibration datasets and several benchmarks. Across the tested conditions, HOPE achieved the best average rank among five methods; at 50% pruning, its average rank was 1.58 versus 2.42 for the next-best method. On Qwen3.5-35B-A3B at that pruning rate, the authors report a score of 42.3 on SWE-Bench Pro, compared with 41.8 for the unpruned model and 36.2 for REAP. The authors describe the gains as most pronounced at 40–50% pruning, while first-order methods are sometimes competitive at 10–20%. HOPE accounts for pairwise interactions within each layer, not cross-layer interactions; the authors say their cross-layer results remain preliminary. How would HOPE’s results change if it accounted for cross-layer interactions or interactions among three or more experts? ## Original source Hugging Face · Article Original source date: Nov 9, 2026 (2026-11-09T12:34:29.000Z). Date basis: Date supplied by the publisher’s feed. Added to Escape Meridian: 2026-11-10T17:03:59.661Z. First observed: 2026-11-09T12:36:33.153Z. Retrieved: 2026-11-09T12:45:40.946Z. - [Your Experts Have Partners. There's HOPE for Keeping Them Together](https://huggingface.co/blog/amazon/higher-order-pruning-experts) ## Limits The reported results cover three models, two calibration datasets and the benchmarks described in the post; they do not establish performance beyond those evaluations. The authors say HOPE's advantage shrinks at lower pruning rates, where first-order methods are sometimes competitive. HOPE does not account for cross-layer interactions, and the authors describe their cross-layer results as preliminary. HOPE models pairwise interactions; the post says interactions among three or more experts could provide additional signal but are difficult to estimate. ## Open question How would HOPE’s results change if it accounted for cross-layer interactions or interactions among three or more experts? ## Source-stated Alex M. Tseng presents HOPE, a pruning method that accounts for pairwise expert interactions in Mixture-of-Experts models. > Our newest paper derives HOPE (Higher-Order Pruning of Experts), a second-order method that accounts for pairwise expert interactions. - [Supporting source](https://huggingface.co/blog/amazon/higher-order-pruning-experts) ## Source-stated The authors report HOPE achieved the best average rank across the tested models, calibration sets and benchmarks. > Across all conditions, HOPE achieved the best average rank (2.07 out of 5 methods) - [Supporting source](https://huggingface.co/blog/amazon/higher-order-pruning-experts) ## Source-stated At 50% pruning, the authors report HOPE's average rank was 1.58, compared with 2.42 for REAP. > At 50% pruning, HOPE's average rank improves to 1.58 (vs. 2.42 for the next-best method, REAP). - [Supporting source](https://huggingface.co/blog/amazon/higher-order-pruning-experts) ## Source-stated On Qwen3.5-35B-A3B at 50% pruning, the authors report HOPE scored 42.3 on SWE-Bench Pro, versus 41.8 for the unpruned model and 36.2 for REAP. > On Qwen3.5-35B-A3B at 50% pruning, HOPE achieved 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8 (while REAP dropped to 36.2, a 13% regression). - [Supporting source](https://huggingface.co/blog/amazon/higher-order-pruning-experts) ## Source-stated The authors say HOPE's advantage is most pronounced at 40–50% pruning; at 10–20%, first-order methods are sometimes competitive. > HOPE's advantage is most pronounced at higher pruning rates (40–50%). At lower pruning rates (10–20%), first-order methods like REAP and MAN are sometimes competitive - [Supporting source](https://huggingface.co/blog/amazon/higher-order-pruning-experts) ## Source-stated HOPE prunes layers independently and does not account for cross-layer interactions; the authors describe their cross-layer results as preliminary. > HOPE prunes each layer independently and does not account for cross-layer interactions. In our paper, we presented a theoretical framework for cross-layer HOPE, but the results remain preliminary. - [Supporting source](https://huggingface.co/blog/amazon/higher-order-pruning-experts) ## Source-stated HOPE captures pairwise interactions, while interactions among three or more experts could provide additional signal, according to the authors. > Finally, HOPE captures pairwise interactions, but even higher-order interactions (triplets and beyond) could provide additional signal - [Supporting source](https://huggingface.co/blog/amazon/higher-order-pruning-experts) ## Source and contributor context Hugging Face · Publisher The publisher is identified here. No individual author has been inferred. ## Source media Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Mixture of experts A model design that sends each input to a few specialized sub-networks instead of the whole model, which saves computation. - [Definition reference](https://developers.google.com/machine-learning/glossary#mixture-of-experts) ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Hugging Face Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems. - [Explore this context](/entities/hugging-face) - [Reference](https://huggingface.co/blog) ## Editorial provenance Version 1. Publication ID: 744e4ab7-f45d-498a-8a6b-9461b7e96541. Prepared by gpt-6-luna; stream-draft-repair-v1; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T17:03:59.661Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: b65c78caee9e22b49fc9227c418eaf1caca12e369ca29818a159c67d4222911f. Source text hash: 74d15c233a49ec339d9b1eee5a02633ec8c8406e33d7c881b3532315f4c67f61. ### uncertain — 2026-11-10T17:03:59.661Z — anthropic/claude-opus-5.5 Publication: 744e4ab7-f45d-498a-8a6b-9461b7e96541 attribution — pass: The post appears under Amazon on Hugging Face, with Alex M Tseng as author. The paper is multi-author, so presenting HOPE through Tseng and reporting results as 'the authors report' is consistent with the source. > Our newest paper derives HOPE (Higher-Order Pruning of Experts), a second-order method that accounts for pairwise expert interactions. summary — uncertain: The draft is faithful but does not say why expert pruning matters. It omits the post's central motivation: every expert must stay in memory even though only a few are used per token. It also leaves 'experts', 'pruning' (permanently deleting experts) and 'first-order' unexplained for non-specialists. Smallest repair: add one sentence on the memory bottleneck and what pruning does, plus term-library tooltips for MoE, expert, pruning, first-order and SWE-Bench Pro. > every expert MLP must occupy memory at inference time, even though the vast majority are idle for any given token date — pass: originalPublished of 2026-11-09 matches the post's stated publication date. The arXiv paper may predate it, but the source under review is this blog post. > Published November 9, 2026 support — pass: All numeric claims, the conditions and the limitations match the source and are attributed. The draft's neutral 'compared with' avoids overstating the source's 'matching' claim. The closing question follows from the stated limitations and implies no unsupported benefit. > Finally, HOPE captures pairwise interactions, but even higher-order interactions (triplets and beyond) could provide additional signal, although they are combinatorially difficult to estimate. media — pass: The selected media is the verified direct link to the original Hugging Face post, the most relevant way to access the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/ab90bf4c83956c6de48ff82f4b5aecde8912e50425298db52fe1bf38be68c4cd/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/your-experts-have-partners-there-s-hope-for-keeping-them-ab90bf4c.json)