arXiv: Zou and Zhong propose KVBuffer for linear attention serving

arXiv: Zou and Zhong propose KVBuffer for linear attention serving

Longwei Zou and Lin Zhong argue that serving linear attention can become inefficient even though its decoding cost stays constant as context length grows. Systems repeatedly access and update a large linear attention state, which is much larger than a token’s key and value. Their proposed KVBuffer buffers recent keys and values so serving systems can choose how to compute outputs. In decoding, it groups work into chunks and defers state updates. The authors say this reduces average memory access and decoding latency. For speculative decoding, it verifies draft tokens in parallel without storing temporary states; for short contexts, it computes outputs from buffered keys and values without creating or updating the state. The authors implemented KVBuffer in SGLang for Qwen3.5 and Nemotron-3 series models. They report up to 1.47x faster batch decoding and a fivefold increase in the maximum number of serving requests for speculative decoding when verifying four draft tokens. These are reported evaluation results for the named setup, not a general guarantee for other models or serving conditions.

Publication: b9cd6444-3dd4-4d18-8d8c-35737ef9cd6f

attribution — pass: The reading correctly credits KVBuffer to Longwei Zou and Lin Zhong on arXiv. Mechanisms and results are framed as the authors' proposal and reported evaluations, not as established fact.

Authors:Longwei Zou, Lin Zhong

summary — uncertain: The summary is faithful to the abstract but hard for a non-specialist to follow. 'Linear attention', 'keys and values', 'decoding', 'speculative decoding', 'draft tokens' and 'SGLang' are never framed in plain words. It never says that serving means running a model to answer user requests, so readers cannot see why lower latency and more concurrent requests matter. Smallest repair: one plain-language framing sentence, keeping the exact terms with tooltips, stating the source-supported stake of faster, higher-capacity serving for these long-context models.

Linear attention has recently gained significant attention for long-context inference due to its constant decoding cost with respect to context length.

date — pass: originalPublished 2026-05-18 matches the v1 submission date. It correctly ignores the 5 Nov 2026 v2 revision and the November feed date.

Submitted on 18 May 2026 (v1), last revised 5 Nov 2026 (this version, v2)

support — pass: All claims match the abstract: the state is larger than per-token keys and values, chunkwise computation defers state updates, speculative verification runs in parallel, short contexts skip the state, and the implementation is in SGLang for Qwen3.5 and Nemotron-3. The 1.47x and 5x figures keep their conditions. The limitations and question appropriately mark scope as unknown without inventing settings.

Our evaluations show that KVBuffer can speedup batch decoding by up to 1.47x and increase the maximum number of serving requests by 5x for speculative decoding when verifying four draft tokens.

media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most direct way to reach the source.