Morisetty, Suhas Kumar, Lokegaonkar, Vaibhavi, Varghese, Sachin Mathew, Harris, Wesley · Oct 26, 2026
arXiv cs.AI: WAKMan, a workload-aware KV cache manager
Markdown · JSON · Original source
The authors of WAKMan argue that no single KV cache eviction policy works best across different mixes of language-model requests. Prefix caching reuses stored Key-Value (KV) states when requests share a prefix, reducing repeated prefill computation; eviction policy determines which states remain in limited memory. The authors report that steps tend to return within seconds and favor recency, while popular prompts and long conversations may return after many other requests and favor frequency. Across 20 mixes, they say the default LRU policy had 12% more cache hits than S3FIFO on agentic-heavy mixes, but up to 22% fewer elsewhere; even one workload parameter could change which policy performed best. WAKMan selects an eviction policy from live traffic without identifying request types or reading prompts. The authors report a 5-percentage-point average hit-rate improvement over LRU, along with the lowest mean and worst-case optimality gap among the policies they tested. These are the paper’s reported results; the abstract does not establish how the approach performs in a particular production deployment. How does WAKMan perform on live production traffic compared with the workloads used in the paper’s evaluation?
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 26, 2026 (2026-10-26).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-14T00:18:24.260Z. First observed: 2026-10-28T08:02:48.017Z. Retrieved: 2026-10-30T06:17:52.948Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The supplied evidence is the paper’s abstract, which does not describe the evaluation setup or provide details needed to assess how the reported results transfer to production deployments.
Open question
How does WAKMan perform on live production traffic compared with the workloads used in the paper’s evaluation?
Source-stated
The authors say prefix caching reuses KV states for requests that share a prefix, reducing prefill computation, and that eviction policy matters when memory is limited.
Prefix caching enables a Large Language Model (LLM) servers to reuse the Key-Value (KV) states and accelerate prefill computation stage for requests that share a prefix with earlier prompts, thus reducing latency and serving cost.
Source-stated
The authors report that agentic steps tend to return within seconds and favor recency, while popular prompts and long conversations favor frequency.
agentic steps return within seconds and favor recency, while popular prompts and long conversations return after many other requests and favor frequency.
Source-stated
Across 20 mixes, the authors report that LRU had 12% more cache hits than S3FIFO on agentic-heavy mixes but up to 22% fewer elsewhere; a single workload parameter could change which policy won.
Across 20 mixes, no single policy is always best: vLLM's default strategy LRU provides 12% higher cache hits as compared to the frequency-oriented S3FIFO on agentic-heavy mixes but trails it by up to 22% elsewhere, and even a single workload parameter can flip the winner even for any particular mix.
Source-stated
WAKMan selects its eviction policy from live traffic without knowing request types or reading prompts.
We present WAKMan, a workload-aware KV cache manager that picks its eviction policy from live traffic without knowing request types or reading prompts.
Source-stated
The authors report that WAKMan improves hit rate over LRU by 5 percentage points on average and has the lowest mean and worst-case optimality gap among the policies.
It improves hit rate over LRU by 5 percentage points on average, with the lowest mean and worst-case optimality gap of all policies.
Source and contributor context
Morisetty, Suhas Kumar, Lokegaonkar, Vaibhavi, Varghese, Sachin Mathew, Harris, Wesley · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Agentic AI
AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 22b76062-8d69-433d-8e12-9612e5107189.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-14T00:18:24.260Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 9c2f3da2ae3aa8d0b2a32340c554156a3197426705c84f2708fb03c68fe9d94d. Source text hash: fccba0bed4d0678e5772af6209d1ec7d8a99dec8f1e769e3fc01e6ee4c7546a1.
Revision and source-check history
Version 1 · 2026-11-14T00:18:24.260Z
arXiv cs.AI: WAKMan, a workload-aware KV cache manager
The authors of WAKMan argue that no single KV cache eviction policy works best across different mixes of language-model requests. Prefix caching reuses stored Key-Value (KV) states when requests share a prefix, reducing repeated prefill computation; eviction policy determines which states remain in limited memory. The authors report that steps tend to return within seconds and favor recency, while popular prompts and long conversations may return after many other requests and favor frequency. Across 20 mixes, they say the default LRU policy had 12% more cache hits than S3FIFO on agentic-heavy mixes, but up to 22% fewer elsewhere; even one workload parameter could change which policy performed best. WAKMan selects an eviction policy from live traffic without identifying request types or reading prompts. The authors report a 5-percentage-point average hit-rate improvement over LRU, along with the lowest mean and worst-case optimality gap among the policies they tested. These are the paper’s reported results; the abstract does not establish how the approach performs in a particular production deployment. How does WAKMan perform on live production traffic compared with the workloads used in the paper’s evaluation?
Original source
cs.AI · Article
Source details
Original source date: Oct 26, 2026 (2026-10-26).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-14T00:18:24.260Z. First observed: 2026-10-28T08:02:48.017Z. Retrieved: 2026-10-30T06:17:52.948Z.
Limits
The supplied evidence is the paper’s abstract, which does not describe the evaluation setup or provide details needed to assess how the reported results transfer to production deployments.
Open question
How does WAKMan perform on live production traffic compared with the workloads used in the paper’s evaluation?
Source-stated
The authors say prefix caching reuses KV states for requests that share a prefix, reducing prefill computation, and that eviction policy matters when memory is limited.
Prefix caching enables a Large Language Model (LLM) servers to reuse the Key-Value (KV) states and accelerate prefill computation stage for requests that share a prefix with earlier prompts, thus reducing latency and serving cost.
Source-stated
The authors report that agentic steps tend to return within seconds and favor recency, while popular prompts and long conversations favor frequency.
agentic steps return within seconds and favor recency, while popular prompts and long conversations return after many other requests and favor frequency.
Source-stated
Across 20 mixes, the authors report that LRU had 12% more cache hits than S3FIFO on agentic-heavy mixes but up to 22% fewer elsewhere; a single workload parameter could change which policy won.
Across 20 mixes, no single policy is always best: vLLM's default strategy LRU provides 12% higher cache hits as compared to the frequency-oriented S3FIFO on agentic-heavy mixes but trails it by up to 22% elsewhere, and even a single workload parameter can flip the winner even for any particular mix.
Source-stated
WAKMan selects its eviction policy from live traffic without knowing request types or reading prompts.
We present WAKMan, a workload-aware KV cache manager that picks its eviction policy from live traffic without knowing request types or reading prompts.
Source-stated
The authors report that WAKMan improves hit rate over LRU by 5 percentage points on average and has the lowest mean and worst-case optimality gap among the policies.
It improves hit rate over LRU by 5 percentage points on average, with the lowest mean and worst-case optimality gap of all policies.
Source and contributor context
Morisetty, Suhas Kumar, Lokegaonkar, Vaibhavi, Varghese, Sachin Mathew, Harris, Wesley · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Agentic AI
AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 22b76062-8d69-433d-8e12-9612e5107189.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-14T00:18:24.260Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 9c2f3da2ae3aa8d0b2a32340c554156a3197426705c84f2708fb03c68fe9d94d. Source text hash: fccba0bed4d0678e5772af6209d1ec7d8a99dec8f1e769e3fc01e6ee4c7546a1.
uncertain · 2026-11-14T00:18:24.260Z · anthropic/claude-opus-5.5
Publication: 22b76062-8d69-433d-8e12-9612e5107189
attribution — pass: The reading attributes the claims to the WAKMan paper's authors on arXiv cs.AI and keeps the results framed as reported findings. That matches the abstract and the listed authors.
Authors:Suhas Kumar Morisetty, Vaibhavi Lokegaonkar, Sachin Mathew Varghese, Wesley Harris
summary — uncertain: The draft says 'the default LRU policy' but drops that LRU is vLLM's default, which leaves 'default' unanchored. Repair: write 'vLLM's default LRU policy'. LRU, S3FIFO, vLLM and 'optimality gap' are central but unexplained, so a general reader cannot follow the comparison. They need term-library tooltips. The draft also omits the source's stated stake, that prefix caching reduces latency and serving cost. Repair: add that phrase. The rest of the reading is faithful and does not overstate.
vLLM's default strategy LRU provides 12% higher cache hits as compared to the frequency-oriented S3FIFO
date — pass: The originalPublished date of 2026-10-26 matches the v1 submission date shown on the arXiv page. The later feed date is correctly not used.
[Submitted on 26 Oct 2026]
support — pass: Each claim matches the abstract. The 12% and up-to-22% figures, the 20 mixes and the 5-percentage-point gain are reported accurately. The limitation and the question about production traffic are appropriately conditional and do not invent benefits or costs.
It improves hit rate over LRU by 5 percentage points on average, with the lowest mean and worst-case optimality gap of all policies.
media — pass: The selected media is the verified direct link to the arXiv abstract page. That page is the original source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivAI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv