Ranjan, Ekagra, Teru, Komal K., Venkitesh, Bharat, Locatelli, Acyr · Oct 28, 2026
Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models
Markdown · JSON · Original source
Authors Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh and Acyr Locatelli argue that speculative decoding can speed up sparse models more at low batch sizes than prior predictions suggest. Speculative decoding drafts several tokens and checks them in one pass; MoE models route work to selected expert networks. The authors evaluate 10 models and report that recent fine-grained models show higher speedup at low batch size, contrary to the predicted pattern of gains peaking at intermediate batch sizes. They attribute the difference to verification costs: repeated reads can be absorbed by L2, bandwidth use differs between MoE layers and attention, and draft tokens may route to overlapping experts. Their model uses routing-overlap statistics to predict how many unique experts verification activates. The authors report less than 4% error and say it narrows the gap to measured model curves. This offers a possible way to better estimate inference speedups, but the abstract does not establish how the findings translate to other hardware or deployments. Do these results hold beyond the evaluated models and profiling?
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 28, 2026 (2026-10-28).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T19:31:58.625Z. First observed: 2026-10-30T06:45:33.877Z. Retrieved: 2026-10-30T16:32:27.669Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The available evidence is the paper's abstract; it does not provide the experimental setup, detailed results, or the full basis for comparing predicted and observed speedups. The reported kernel profiling is specifically described for ; the abstract does not establish whether the findings generalize to other hardware or deployments. The abstract reports evaluation across sparse models but does not identify their names or give model-by-model results.
Open question
Do these results hold beyond the evaluated models and profiling?
Source-stated
The authors report evaluating 10 sparse MoE models and finding recent fine-grained models show higher speculative-decoding speedup at low batch sizes, contrary to prior predictions.
We evaluate this across 10 sparse MoE models (30B--1T parameters, 8--512 experts) and find recent fine-grained models break this trend, showing higher speedup at low batch size.
Source-stated
The authors say verification costs depend on more than the bytes of activated experts, including draft-token routing overlap and GPU execution dynamics.
We find verification cost is governed not only by activated-expert bytes, but also by temporal overlap across draft tokens and GPU execution dynamics.
Source-stated
For low batch sizes, the authors attribute a smaller-than-expected runtime share for MoE layers partly to GPU bandwidth use and L2 cache behavior on H100.
on H100, kernel profiling attributes this mainly to higher bandwidth utilization of MoE kernels relative to attention, with L2 absorbing repeated reads of co-routed experts.
Source-stated
The authors report that their Iterative Proportional Fitting framework predicts unique expert counts with less than 4% error.
We model this effect using a maximum-entropy Iterative Proportional Fitting (IPF) framework that predicts unique expert counts from routing-overlap statistics with less than 4\% error
Source and contributor context
Ranjan, Ekagra, Teru, Komal K., Venkitesh, Bharat, Locatelli, Acyr · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Mixture of experts
A model design that sends each input to a few specialized sub-networks instead of the whole model, which saves computation.
Graphics processing unit
A chip that performs many calculations at once; the main hardware used to train and run AI models.
NVIDIA H100
An NVIDIA graphics processing unit (GPU) built on the company's Hopper architecture and designed to accelerate AI workloads, including large language models.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 44732b85-407f-426d-8717-832e13ad9bf2.
Prepared by server-excerpt-repair/stream-excerpt-repair-v1 (original draft: gpt-6-luna); grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T19:31:58.625Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: d5ba7a50062210c739c073a95b52dae42149782d289fe246a2bf093bc7b08ed2. Source text hash: 0dd29b6e186d9a7a578fc17ec18b618cdd42a2026b1bf9cb035f3fcb21cdac9d.
Revision and source-check history
Version 1 · 2026-11-10T19:31:58.625Z
Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models
Authors Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh and Acyr Locatelli argue that speculative decoding can speed up sparse models more at low batch sizes than prior predictions suggest. Speculative decoding drafts several tokens and checks them in one pass; MoE models route work to selected expert networks. The authors evaluate 10 models and report that recent fine-grained models show higher speedup at low batch size, contrary to the predicted pattern of gains peaking at intermediate batch sizes. They attribute the difference to verification costs: repeated reads can be absorbed by L2, bandwidth use differs between MoE layers and attention, and draft tokens may route to overlapping experts. Their model uses routing-overlap statistics to predict how many unique experts verification activates. The authors report less than 4% error and say it narrows the gap to measured model curves. This offers a possible way to better estimate inference speedups, but the abstract does not establish how the findings translate to other hardware or deployments. Do these results hold beyond the evaluated models and profiling?
Original source
cs.AI · Article
Source details
Original source date: Oct 28, 2026 (2026-10-28).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T19:31:58.625Z. First observed: 2026-10-30T06:45:33.877Z. Retrieved: 2026-10-30T16:32:27.669Z.
Limits
The available evidence is the paper's abstract; it does not provide the experimental setup, detailed results, or the full basis for comparing predicted and observed speedups. The reported kernel profiling is specifically described for ; the abstract does not establish whether the findings generalize to other hardware or deployments. The abstract reports evaluation across sparse models but does not identify their names or give model-by-model results.
Open question
Do these results hold beyond the evaluated models and profiling?
Source-stated
The authors report evaluating 10 sparse MoE models and finding recent fine-grained models show higher speculative-decoding speedup at low batch sizes, contrary to prior predictions.
We evaluate this across 10 sparse MoE models (30B--1T parameters, 8--512 experts) and find recent fine-grained models break this trend, showing higher speedup at low batch size.
Source-stated
The authors say verification costs depend on more than the bytes of activated experts, including draft-token routing overlap and GPU execution dynamics.
We find verification cost is governed not only by activated-expert bytes, but also by temporal overlap across draft tokens and GPU execution dynamics.
Source-stated
For low batch sizes, the authors attribute a smaller-than-expected runtime share for MoE layers partly to GPU bandwidth use and L2 cache behavior on H100.
on H100, kernel profiling attributes this mainly to higher bandwidth utilization of MoE kernels relative to attention, with L2 absorbing repeated reads of co-routed experts.
Source-stated
The authors report that their Iterative Proportional Fitting framework predicts unique expert counts with less than 4% error.
We model this effect using a maximum-entropy Iterative Proportional Fitting (IPF) framework that predicts unique expert counts from routing-overlap statistics with less than 4\% error
Source and contributor context
Ranjan, Ekagra, Teru, Komal K., Venkitesh, Bharat, Locatelli, Acyr · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Mixture of experts
A model design that sends each input to a few specialized sub-networks instead of the whole model, which saves computation.
Graphics processing unit
A chip that performs many calculations at once; the main hardware used to train and run AI models.
NVIDIA H100
An NVIDIA graphics processing unit (GPU) built on the company's Hopper architecture and designed to accelerate AI workloads, including large language models.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 44732b85-407f-426d-8717-832e13ad9bf2.
Prepared by server-excerpt-repair/stream-excerpt-repair-v1 (original draft: gpt-6-luna); grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T19:31:58.625Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: d5ba7a50062210c739c073a95b52dae42149782d289fe246a2bf093bc7b08ed2. Source text hash: 0dd29b6e186d9a7a578fc17ec18b618cdd42a2026b1bf9cb035f3fcb21cdac9d.
uncertain · 2026-11-10T19:31:58.625Z · anthropic/claude-opus-5.5
Publication: 44732b85-407f-426d-8717-832e13ad9bf2
attribution — pass: The reading correctly names the four listed authors, Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh and Acyr Locatelli. It keeps the findings attributed to them as reported results rather than established fact.
Authors:Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh, Acyr Locatelli
summary — uncertain: The opening sentence says sparse MoE models in general speed up more at low batch sizes. The source limits that finding to recent fine-grained models; smallest repair: say 'recent fine-grained sparse MoE models'. Batch size, the central variable, is never explained (roughly, how many requests are processed together). Fine-grained, L2 and H100 also lack plain-language context. Needed term-library tooltips: batch size, fine-grained MoE, L2 cache, H100. Without these, a non-specialist cannot follow why low versus intermediate batch size matters.
find recent fine-grained models break this trend, showing higher speedup at low batch size
date — pass: originalPublished is 2026-10-28, which matches the v1 submission date on arXiv. The later feed date is correctly not used.
[Submitted on 28 Oct 2026]
support — uncertain: Claim 3 says the authors attribute the smaller MoE runtime share 'partly' to bandwidth use and L2 behavior; the source says 'mainly'. This weakens the authors' stated attribution, so the smallest repair is to replace 'partly' with 'mainly'. The summary also drops 'substantially' from 'substantially narrows the gap', mildly understating the claim. The closing inference about better estimating speedups is hedged and acceptable. The question about H100 and other models follows from the evidence.
on H100, kernel profiling attributes this mainly to higher bandwidth utilization of MoE kernels relative to attention
media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A model design that sends each input to a few specialized sub-networks instead of the whole model, which saves computation.
Definition referenceA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition referenceAn NVIDIA graphics processing unit (GPU) built on the company's Hopper architecture and designed to accelerate AI workloads, including large language models.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivAn NVIDIA graphics processing unit (GPU) built on the company's Hopper architecture and designed to accelerate AI workloads, including large language models.
Definition referenceA model design that sends each input to a few specialized sub-networks instead of the whole model, which saves computation.
Definition referenceAn NVIDIA graphics processing unit (GPU) built on the company's Hopper architecture and designed to accelerate AI workloads, including large language models.
Definition referenceA model design that sends each input to a few specialized sub-networks instead of the whole model, which saves computation.
Definition referenceA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition referenceAn NVIDIA graphics processing unit (GPU) built on the company's Hopper architecture and designed to accelerate AI workloads, including large language models.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivAn NVIDIA graphics processing unit (GPU) built on the company's Hopper architecture and designed to accelerate AI workloads, including large language models.
Definition referenceA model design that sends each input to a few specialized sub-networks instead of the whole model, which saves computation.
Definition referenceAn NVIDIA graphics processing unit (GPU) built on the company's Hopper architecture and designed to accelerate AI workloads, including large language models.
Definition reference