# arXiv cs.AI paper argues brief bursts can leave agent serving degraded [Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-paper-argues-brief-bursts-can-leave-agent-serving-799b397b) Wen Huang and Jack Liu argue that prefix-cached agent serving can enter a metastable degraded state: a short burst of arrivals may leave a server struggling after the burst ends. In their tests, slower service kept more sessions in the system, expanded their working set beyond the KV pool, and turned each turn into recomputation. With identical base workloads, 9 of 12 paired vLLM runs with a burst degraded, while none of their no-burst twins did; throughput fell to about half, and the backlog grew for the remaining 70 minutes. The authors report similar behavior with Qwen3-32B on SWE-bench agent trace shapes and with SGLang, though SGLang needed larger triggers. They report that throttling new admissions recovered only 1 of 7 matched collapsed runs, while suspending sessions already admitted against a capacity-based limit recovered all. A budgeter limiting active sessions' resident KV demand to the pool prevented failure in every run they tested. These findings describe the tested systems and workloads; the paper's abstract does not establish how broadly they apply to production deployments. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Nov 10, 2026 (2026-11-10). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-12T03:16:41.524Z. First observed: 2026-11-11T05:09:26.268Z. Retrieved: 2026-11-12T03:01:31.643Z. - [Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving](https://arxiv.org/abs/2611.13412) ## Limits The evidence provided is the paper's abstract, not its full methods or results; the reported findings are limited to the systems, workloads, and runs described there. ## Open question How well does the reported capacity-based budgeter prevent this degradation on agent-serving workloads beyond those tested? ## Source-stated Huang and Liu report that brief extra arrivals can push prefix-cached agent serving into a degraded state that persists after the burst. > a few minutes of extra arrivals leaves a vLLM server in a degraded regime - [Supporting source](https://arxiv.org/abs/2611.13412) ## Source-stated In paired runs with identical base workloads, 9 of 12 burst runs degraded, while none of the no-burst twins did. > (9 of 12 pairs; none in the reverse direction) - [Supporting source](https://arxiv.org/abs/2611.13412) ## Source-stated The authors attribute persistence to slower service keeping more sessions active, growing the working set beyond the KV pool, and causing recomputation on every turn. > Slower service keeps more sessions in the system, their working set outgrows the KV pool, and every turn becomes a recomputation - [Supporting source](https://arxiv.org/abs/2611.13412) ## Source-stated In matched collapsed runs, suspending admitted sessions against a capacity-based limit recovered all runs within minutes; throttling new admissions recovered 1 of 7. > throttling new admissions recovered 1 of 7 matched collapsed runs, while suspending admitted sessions against a capacity-based limit recovered all of them within minutes - [Supporting source](https://arxiv.org/abs/2611.13412) ## Source-stated A budgeter bounding active sessions' resident KV demand to the pool prevented failure in every tested run. > A budgeter that bounds the resident KV demand of active sessions to the pool, with thresholds that need no per-workload tuning, prevented the failure in every run we tested. - [Supporting source](https://arxiv.org/abs/2611.13412) ## Source and contributor context Huang, Wen, Liu, Jack · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 18cc0bb0-81b1-4f15-82f8-f92bbe2007a6. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T03:16:41.524Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: cc81134bb887ba07d2987f5c330bdef5d8eeceb1e23deaf9d13f6c9d296c7339. Source text hash: 030a897f7d1ea8bd4b065795b3d52bbf9082d8efb81428a92c569f8500c0d921. ## Revision and source-check history ### Version 1 — 2026-11-12T03:16:41.524Z Wen Huang and Jack Liu argue that prefix-cached agent serving can enter a metastable degraded state: a short burst of arrivals may leave a server struggling after the burst ends. In their tests, slower service kept more sessions in the system, expanded their working set beyond the KV pool, and turned each turn into recomputation. With identical base workloads, 9 of 12 paired vLLM runs with a burst degraded, while none of their no-burst twins did; throughput fell to about half, and the backlog grew for the remaining 70 minutes. The authors report similar behavior with Qwen3-32B on SWE-bench agent trace shapes and with SGLang, though SGLang needed larger triggers. They report that throttling new admissions recovered only 1 of 7 matched collapsed runs, while suspending sessions already admitted against a capacity-based limit recovered all. A budgeter limiting active sessions' resident KV demand to the pool prevented failure in every run they tested. These findings describe the tested systems and workloads; the paper's abstract does not establish how broadly they apply to production deployments. ## Original source arXiv cs.AI · Article Original source date: Nov 10, 2026 (2026-11-10). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-12T03:16:41.524Z. First observed: 2026-11-11T05:09:26.268Z. Retrieved: 2026-11-12T03:01:31.643Z. - [Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving](https://arxiv.org/abs/2611.13412) ## Limits The evidence provided is the paper's abstract, not its full methods or results; the reported findings are limited to the systems, workloads, and runs described there. ## Open question How well does the reported capacity-based budgeter prevent this degradation on agent-serving workloads beyond those tested? ## Source-stated Huang and Liu report that brief extra arrivals can push prefix-cached agent serving into a degraded state that persists after the burst. > a few minutes of extra arrivals leaves a vLLM server in a degraded regime - [Supporting source](https://arxiv.org/abs/2611.13412) ## Source-stated In paired runs with identical base workloads, 9 of 12 burst runs degraded, while none of the no-burst twins did. > (9 of 12 pairs; none in the reverse direction) - [Supporting source](https://arxiv.org/abs/2611.13412) ## Source-stated The authors attribute persistence to slower service keeping more sessions active, growing the working set beyond the KV pool, and causing recomputation on every turn. > Slower service keeps more sessions in the system, their working set outgrows the KV pool, and every turn becomes a recomputation - [Supporting source](https://arxiv.org/abs/2611.13412) ## Source-stated In matched collapsed runs, suspending admitted sessions against a capacity-based limit recovered all runs within minutes; throttling new admissions recovered 1 of 7. > throttling new admissions recovered 1 of 7 matched collapsed runs, while suspending admitted sessions against a capacity-based limit recovered all of them within minutes - [Supporting source](https://arxiv.org/abs/2611.13412) ## Source-stated A budgeter bounding active sessions' resident KV demand to the pool prevented failure in every tested run. > A budgeter that bounds the resident KV demand of active sessions to the pool, with thresholds that need no per-workload tuning, prevented the failure in every run we tested. - [Supporting source](https://arxiv.org/abs/2611.13412) ## Source and contributor context Huang, Wen, Liu, Jack · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 18cc0bb0-81b1-4f15-82f8-f92bbe2007a6. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T03:16:41.524Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: cc81134bb887ba07d2987f5c330bdef5d8eeceb1e23deaf9d13f6c9d296c7339. Source text hash: 030a897f7d1ea8bd4b065795b3d52bbf9082d8efb81428a92c569f8500c0d921. ### uncertain — 2026-11-12T03:16:41.524Z — anthropic/claude-opus-5.5 Publication: 18cc0bb0-81b1-4f15-82f8-f92bbe2007a6 attribution — pass: The reading attributes the paper to Wen Huang and Jack Liu on arXiv, matching the source. The paper's primary category is cs.DC with a cs.AI cross-listing, so the title's 'arXiv cs.AI paper' is acceptable. Findings are consistently framed as the authors' reports. > Authors:Wen Huang, Jack Liu summary — uncertain: Central jargon is left unexplained for non-specialists: prefix caching, KV pool, prefill, metastable, vLLM and SGLang. The reading also omits the abstract's key cost asymmetry, which explains why cache misses are so damaging. It drops several conditions too: the regime appears at loads a cache-warm server sustains, it is absent without prefix caching, a CPU offload tier only delays collapse, and the budgeter's thresholds need no per-workload tuning. Smallest repair: add the hit-versus-miss cost sentence with plain-language glosses and the no-prefix-caching condition. > a hit costs an incremental prefill of about a thousand tokens, a miss recomputes a 10--40K-token context date — pass: originalPublished is 2026-11-10, which matches the v1 submission date. The draft makes no conflicting date claim. > \[Submitted on 10 Nov 2026\] support — uncertain: The source says that in 9 of 12 pairs the burst run degraded while its no-burst twin stayed healthy, with none in the reverse direction. That does not establish that exactly 9 burst runs degraded, or that no no-burst twin ever degraded: the other 3 pairs are unspecified. The summary and the second claim should be rephrased to the source's pair-based framing. The question's 'capacity-based budgeter' also blends the capacity-based suspension mechanism with the separately described budgeter. Other claims are grounded. > while its no-burst twin stays healthy (9 of 12 pairs; none in the reverse direction) media — pass: The selected media is the verified direct link to the arXiv abstract page of this paper, which is the most direct access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/799b397bf6b0b60a90561ef72cb3bca799120c0154d0dbe4cf8b010d9a6baba0/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-paper-argues-brief-bursts-can-leave-agent-serving-799b397b.json)