Malvankar, Abhishek, Castro, Paul, Mitchell, Nick, Villard, Lionel · Oct 28, 2026
arXiv cs.AI: TableTune measures and adjusts draft-length tables for speculative decoding
Markdown · JSON · Original source
Abhishek Malvankar, Paul Castro, Nick Mitchell and Lionel Villard present TableTune, a system for choosing and adjusting draft lengths in speculative decoding. A draft length is how many tokens a small drafter guesses at each step; the paper says the best length shrinks as the batch grows, while vLLM’s hand-written table does not measure or correct itself. TableTune times decoding steps across batch sizes and draft lengths, then builds a table intended to maximize throughput. Its controller tracks accepted guesses in live traffic and rebuilds the table while serving. The authors report that, on an , the tuned table reached at least 94% of the throughput of the best fixed draft length at every load in four of five drafter settings, and 88% in the fifth. They also report gains in tests involving mistuned tables, workload changes and an llm-d deployment. These results suggest that measuring and adapting the table could reduce the need for operators to know the right draft length in advance. The evidence is limited to the reported comparisons and deployments; the abstract does not establish how performance varies across other hardware or workloads. How well does TableTune perform on workloads and hardware beyond those reported in the paper?
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 28, 2026 (2026-10-28).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-07T06:00:59.261Z. First observed: 2026-10-30T06:42:52.486Z. Retrieved: 2026-10-31T09:31:38.638Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The reported results compare TableTune with other ways to speculate and are tied to the paper's tested settings; the abstract does not establish performance on other hardware or workloads. The publisher-page evidence contains the abstract, not the full paper, so details of the experimental setup and evaluation are unavailable here.
Open question
How well does TableTune perform on workloads and hardware beyond those reported in the paper?
Source-stated
The authors say the best draft length shrinks as batch size grows, while vLLM lets operators specify a hand-written table that does not measure or correct itself.
the best draft length, the number of tokens a small drafter guesses per step, shrinks as the batch grows. vLLM lets operators set the draft length for each batch size with a hand-written table, but nothing measures or corrects that table.
Source-stated
TableTune times decoding steps for each batch size and draft length, computes a table intended to maximize throughput, and adjusts it during serving using observed accepted guesses.
Its tuner times the decoding step for every batch size and draft length on vLLM's own serving path and computes, in minutes, the table that maximizes throughput. Its controller, a vLLM plugin, tracks how many guesses are accepted in live traffic and rebuilds the table while serving.
Source-stated
On an H100, the authors report that the tuned table reached at least 94% of the best fixed draft length's throughput at every load in four of five drafter settings, and 88% in the fifth.
On an H100, the tuned table delivers at least 94% of the throughput of the best fixed draft length at every load in four of five drafter settings (88% in the fifth)
Source-stated
The paper reports throughput improvements from adjusting mistuned tables and after a workload switch, as well as gains in an llm-d deployment.
Starting from mistuned tables, the controller raises throughput by up to 58% (up to 45% on vLLM's default runner), and after a workload switch with the DFlash drafter it recovers 96% of a stale table's loss.
Source and contributor context
Malvankar, Abhishek, Castro, Paul, Mitchell, Nick, Villard, Lionel · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
NVIDIA H100
An NVIDIA graphics processing unit (GPU) built on the company's Hopper architecture and designed to accelerate AI workloads, including large language models.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 919cf774-3928-479b-803e-3d29acc5197e.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-07T06:00:59.261Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 383f59c058871f9f301608607500b510578516ef0c00d556f2d8bed8ef38eb62. Source text hash: d5ca104dd139871e26298266caf0b2630dc19f0e0b9f892ad54d913f6d12419f.
Revision and source-check history
Version 1 · 2026-11-07T06:00:59.261Z
arXiv cs.AI: TableTune measures and adjusts draft-length tables for speculative decoding
Abhishek Malvankar, Paul Castro, Nick Mitchell and Lionel Villard present TableTune, a system for choosing and adjusting draft lengths in speculative decoding. A draft length is how many tokens a small drafter guesses at each step; the paper says the best length shrinks as the batch grows, while vLLM’s hand-written table does not measure or correct itself. TableTune times decoding steps across batch sizes and draft lengths, then builds a table intended to maximize throughput. Its controller tracks accepted guesses in live traffic and rebuilds the table while serving. The authors report that, on an , the tuned table reached at least 94% of the throughput of the best fixed draft length at every load in four of five drafter settings, and 88% in the fifth. They also report gains in tests involving mistuned tables, workload changes and an llm-d deployment. These results suggest that measuring and adapting the table could reduce the need for operators to know the right draft length in advance. The evidence is limited to the reported comparisons and deployments; the abstract does not establish how performance varies across other hardware or workloads. How well does TableTune perform on workloads and hardware beyond those reported in the paper?
Original source
cs.AI · Article
Source details
Original source date: Oct 28, 2026 (2026-10-28).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-07T06:00:59.261Z. First observed: 2026-10-30T06:42:52.486Z. Retrieved: 2026-10-31T09:31:38.638Z.
Limits
The reported results compare TableTune with other ways to speculate and are tied to the paper's tested settings; the abstract does not establish performance on other hardware or workloads. The publisher-page evidence contains the abstract, not the full paper, so details of the experimental setup and evaluation are unavailable here.
Open question
How well does TableTune perform on workloads and hardware beyond those reported in the paper?
Source-stated
The authors say the best draft length shrinks as batch size grows, while vLLM lets operators specify a hand-written table that does not measure or correct itself.
the best draft length, the number of tokens a small drafter guesses per step, shrinks as the batch grows. vLLM lets operators set the draft length for each batch size with a hand-written table, but nothing measures or corrects that table.
Source-stated
TableTune times decoding steps for each batch size and draft length, computes a table intended to maximize throughput, and adjusts it during serving using observed accepted guesses.
Its tuner times the decoding step for every batch size and draft length on vLLM's own serving path and computes, in minutes, the table that maximizes throughput. Its controller, a vLLM plugin, tracks how many guesses are accepted in live traffic and rebuilds the table while serving.
Source-stated
On an H100, the authors report that the tuned table reached at least 94% of the best fixed draft length's throughput at every load in four of five drafter settings, and 88% in the fifth.
On an H100, the tuned table delivers at least 94% of the throughput of the best fixed draft length at every load in four of five drafter settings (88% in the fifth)
Source-stated
The paper reports throughput improvements from adjusting mistuned tables and after a workload switch, as well as gains in an llm-d deployment.
Starting from mistuned tables, the controller raises throughput by up to 58% (up to 45% on vLLM's default runner), and after a workload switch with the DFlash drafter it recovers 96% of a stale table's loss.
Source and contributor context
Malvankar, Abhishek, Castro, Paul, Mitchell, Nick, Villard, Lionel · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
NVIDIA H100
An NVIDIA graphics processing unit (GPU) built on the company's Hopper architecture and designed to accelerate AI workloads, including large language models.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 919cf774-3928-479b-803e-3d29acc5197e.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-07T06:00:59.261Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 383f59c058871f9f301608607500b510578516ef0c00d556f2d8bed8ef38eb62. Source text hash: d5ca104dd139871e26298266caf0b2630dc19f0e0b9f892ad54d913f6d12419f.
uncertain · 2026-11-07T06:00:59.261Z · anthropic/claude-opus-5.5
Publication: 919cf774-3928-479b-803e-3d29acc5197e
attribution — pass: The draft correctly names the four authors and attributes the design, measurements and results to them as reported claims. The 'arXiv cs.AI' label matches the supplied publisher, even though the primary listing is cs.ET.
Authors:Abhishek Malvankar, Paul Castro, Nick Mitchell, Lionel Villard
summary — uncertain: A non-specialist cannot readily understand the central idea. The draft never says in plain language what speculative decoding is: a small drafter guesses tokens that the large model then checks. It also drops the source's causal mechanism, that speculation helps only while the GPU has spare compute, which is why the best length shrinks as batches grow. vLLM, batch and throughput are also unexplained. Smallest repair: add one clause on what speculative decoding does and the spare-compute condition, and identify vLLM as LLM serving software. The question and the other content are faithful.
Speculative decoding speeds up large language model (LLM) serving only while the GPU has spare compute, so the best draft length
date — pass: The originalPublished date of 2026-10-28 matches the v1 submission date shown on the arXiv page. It is not the later feed date.
[Submitted on 28 Oct 2026]
support — pass: The tuner and controller descriptions, the H100 figures (94% and 88%), and the gains with mistuned tables, workload switches and llm-d are all stated in the abstract. The inference that operators need less advance knowledge is conditional and grounded in the source. The limitations correctly note that comparisons are only against other ways to speculate.
without the operator having to know which length that is
media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
An NVIDIA graphics processing unit (GPU) built on the company's Hopper architecture and designed to accelerate AI workloads, including large language models.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivAn NVIDIA graphics processing unit (GPU) built on the company's Hopper architecture and designed to accelerate AI workloads, including large language models.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv