Guo, Yipin, George, Arun M, Joshi, Siddharth · Oct 28, 2026
arXiv cs.AI: GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices
Markdown · JSON · Original source
Yipin Guo, Arun M George and Siddharth Joshi present GAAC, a system for compressing the key-value (KV) cache used during inference. They argue that this cache can become a memory bottleneck for long-context inference on personal devices, even when have been compressed. GAAC encodes BF16 exponents in groups and reconstructs KV tiles as attention runs; the authors say it preserves KV values bit-for-bit rather than approximating them. Across five tested models, they report 1.405–1.407× lossless BF16 KV compression. On DGX Spark, they report faster decode attention at a 64K-token context and 1.04–1.15× faster end-to-end W4A16 decode steps. In serving, they report about 26% less resident KV memory and 13–29% more aggregate throughput under comparable KV-memory budgets. They also report attention speedups on Apple M4, suggesting the approach is not limited to NVIDIA in their tests. Escape Meridian interpretation: the results make the cache itself—not only model weights—a potential target for reducing memory use and speeding inference on personal devices. The abstract does not identify the tested models or give enough experimental detail to assess how broadly the reported gains apply.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 28, 2026 (2026-10-28).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-08T06:16:23.289Z. First observed: 2026-10-30T06:42:45.896Z. Retrieved: 2026-11-08T06:01:40.612Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The abstract does not identify the five tested models or provide enough experimental detail to assess how broadly the reported gains apply. The reported measurements are attributed to the paper's authors; the supplied evidence is the abstract, not an independent evaluation.
Open question
How do GAAC's reported compression and serving results vary across the five tested models and the reported device setups?
Source-stated
The authors introduce GAAC as a lossless KV compression and serving system that encodes BF16 exponents and preserves KV values bit-for-bit.
GAAC encodes BF16 exponents using calibrated per-channel base-plus-offset codes over contiguous ranges.
Source-stated
Across five tested models, the authors report 1.405–1.407× lossless BF16 KV compression.
Across five tested models, GAAC achieves 1.405-1.407$\times$ lossless BF16 KV compression.
Source-stated
On NVIDIA DGX Spark, the authors report decode-attention and end-to-end W4A16 decode-step speedups.
On NVIDIA DGX Spark, GAAC accelerates decode attention at a 64K-token context by 1.19-1.27$\times$ and end-to-end W4A16 decode steps by 1.04-1.15$\times$.
Source-stated
For serving, the authors report reduced resident KV memory and higher aggregate throughput under comparable KV-memory budgets.
In serving, it reduces resident KV memory by approximately 26% and increases aggregate throughput by 13-29% under comparable KV-memory budgets.
Escape Meridian interpretation
The abstract's reported gains make KV-cache compression a potential way to reduce memory use and speed inference on personal devices; the abstract alone does not establish how broadly those gains apply.
the growing key-value (KV) cache remains a bottleneck for long-context inference on personal devices.
Source and contributor context
Guo, Yipin, George, Arun M, Joshi, Siddharth · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Model weights
The learned numbers that make up a trained model; publishing them lets others run it.
Graphics processing unit
A chip that performs many calculations at once; the main hardware used to train and run AI models.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
NVIDIA
NVIDIA develops processors, networking and software used for AI, graphics and scientific computing.
Editorial provenance
Version 1. Publication ID: 02b3e530-1d35-4756-864e-ca0eed88bcde.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-08T06:16:23.289Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 25c66c49eb2502c1cf92281e9dd39227ffd2d1e78c547361dc7c86488026435e. Source text hash: 82cf4d1495c634172dba883c610045427e054e3c0ba89910c727b35dd661d402.
Revision and source-check history
Version 1 · 2026-11-08T06:16:23.289Z
arXiv cs.AI: GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices
Yipin Guo, Arun M George and Siddharth Joshi present GAAC, a system for compressing the key-value (KV) cache used during inference. They argue that this cache can become a memory bottleneck for long-context inference on personal devices, even when have been compressed. GAAC encodes BF16 exponents in groups and reconstructs KV tiles as attention runs; the authors say it preserves KV values bit-for-bit rather than approximating them. Across five tested models, they report 1.405–1.407× lossless BF16 KV compression. On DGX Spark, they report faster decode attention at a 64K-token context and 1.04–1.15× faster end-to-end W4A16 decode steps. In serving, they report about 26% less resident KV memory and 13–29% more aggregate throughput under comparable KV-memory budgets. They also report attention speedups on Apple M4, suggesting the approach is not limited to NVIDIA in their tests. Escape Meridian interpretation: the results make the cache itself—not only model weights—a potential target for reducing memory use and speeding inference on personal devices. The abstract does not identify the tested models or give enough experimental detail to assess how broadly the reported gains apply.
Original source
cs.AI · Article
Source details
Original source date: Oct 28, 2026 (2026-10-28).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-08T06:16:23.289Z. First observed: 2026-10-30T06:42:45.896Z. Retrieved: 2026-11-08T06:01:40.612Z.
Limits
The abstract does not identify the five tested models or provide enough experimental detail to assess how broadly the reported gains apply. The reported measurements are attributed to the paper's authors; the supplied evidence is the abstract, not an independent evaluation.
Open question
How do GAAC's reported compression and serving results vary across the five tested models and the reported device setups?
Source-stated
The authors introduce GAAC as a lossless KV compression and serving system that encodes BF16 exponents and preserves KV values bit-for-bit.
GAAC encodes BF16 exponents using calibrated per-channel base-plus-offset codes over contiguous ranges.
Source-stated
Across five tested models, the authors report 1.405–1.407× lossless BF16 KV compression.
Across five tested models, GAAC achieves 1.405-1.407$\times$ lossless BF16 KV compression.
Source-stated
On NVIDIA DGX Spark, the authors report decode-attention and end-to-end W4A16 decode-step speedups.
On NVIDIA DGX Spark, GAAC accelerates decode attention at a 64K-token context by 1.19-1.27$\times$ and end-to-end W4A16 decode steps by 1.04-1.15$\times$.
Source-stated
For serving, the authors report reduced resident KV memory and higher aggregate throughput under comparable KV-memory budgets.
In serving, it reduces resident KV memory by approximately 26% and increases aggregate throughput by 13-29% under comparable KV-memory budgets.
Escape Meridian interpretation
The abstract's reported gains make KV-cache compression a potential way to reduce memory use and speed inference on personal devices; the abstract alone does not establish how broadly those gains apply.
the growing key-value (KV) cache remains a bottleneck for long-context inference on personal devices.
Source and contributor context
Guo, Yipin, George, Arun M, Joshi, Siddharth · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Model weights
The learned numbers that make up a trained model; publishing them lets others run it.
Graphics processing unit
A chip that performs many calculations at once; the main hardware used to train and run AI models.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
NVIDIA
NVIDIA develops processors, networking and software used for AI, graphics and scientific computing.
Editorial provenance
Version 1. Publication ID: 02b3e530-1d35-4756-864e-ca0eed88bcde.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-08T06:16:23.289Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 25c66c49eb2502c1cf92281e9dd39227ffd2d1e78c547361dc7c86488026435e. Source text hash: 82cf4d1495c634172dba883c610045427e054e3c0ba89910c727b35dd661d402.
passed · 2026-11-08T06:16:23.289Z · anthropic/claude-opus-5.5
Publication: 02b3e530-1d35-4756-864e-ca0eed88bcde
attribution — pass: The draft correctly credits Yipin Guo, Arun M George and Siddharth Joshi as the authors of GAAC. It attributes the bottleneck argument and all measurements to the authors, and it labels the one inference as Escape Meridian interpretation.
Authors:Yipin Guo, Arun M George, Siddharth Joshi
summary — pass: The draft leads with the actor and the development, then gives the authors' motivating problem: the KV cache stays a bottleneck even after weights are compressed. It says what lossless means here (bit-for-bit preservation) and reports the figures faithfully. Terms such as BF16, W4A16 and decode attention are kept exactly, consistent with tooltip policy. The draft softens the source's 'demonstrating portability beyond NVIDIA GPUs' to a test-limited claim, which is cautious but not distorting. The question is substantive and non-leading. It names its gaps, the unidentified models and limited experimental detail, without inventing context.
the growing key-value (KV) cache remains a bottleneck for long-context inference on personal devices
date — pass: originalPublished is 2026-10-28, which matches the v1 submission date. It does not use the later v2 revision (4 Nov) or the feed date (6 Nov).
[Submitted on 28 Oct 2026 (v1), last revised 4 Nov 2026 (this version, v2)]
support — pass: Each numeric claim matches the abstract: 1.405-1.407x compression, 1.04-1.15x decode steps, about 26% less resident KV memory, and 13-29% more aggregate throughput. The bit-for-bit claim is stated in the source. The interpretation is conditional ('potential target') and grounded in the authors' stated bottleneck. The draft notes that the evidence is an abstract, not an independent evaluation, and it does not overclaim real-world benefits.
GAAC adapts code widths across small groups while preserving KV values bit-for-bit.
media — pass: The selected media ID is the verified direct link to the original arXiv abstract page. It is the most direct way to access the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A language model with many learned numerical settings, used to process and generate language.
Definition referenceThe learned numbers that make up a trained model; publishing them lets others run it.
Definition referenceNVIDIA develops processors, networking and software used for AI, graphics and scientific computing.
Explore NVIDIAA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referenceThe learned numbers that make up a trained model; publishing them lets others run it.
Definition referenceNVIDIA develops processors, networking and software used for AI, graphics and scientific computing.
Explore NVIDIAA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv