# arXiv cs.AI: GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices [Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-gaac-lossless-kv-compression-for-bandwidth-e6650ccb) Yipin Guo, Arun M George and Siddharth Joshi present GAAC, a system for compressing the key-value (KV) cache used during large language model inference. They argue that this cache can become a memory bottleneck for long-context inference on personal devices, even when model weights have been compressed. GAAC encodes BF16 exponents in groups and reconstructs KV tiles as attention runs; the authors say it preserves KV values bit-for-bit rather than approximating them. Across five tested models, they report 1.405–1.407× lossless BF16 KV compression. On NVIDIA DGX Spark, they report faster decode attention at a 64K-token context and 1.04–1.15× faster end-to-end W4A16 decode steps. In serving, they report about 26% less resident KV memory and 13–29% more aggregate throughput under comparable KV-memory budgets. They also report attention speedups on Apple M4, suggesting the approach is not limited to NVIDIA GPUs in their tests. Escape Meridian interpretation: the results make the cache itself—not only model weights—a potential target for reducing memory use and speeding inference on personal devices. The abstract does not identify the tested models or give enough experimental detail to assess how broadly the reported gains apply. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Oct 28, 2026 (2026-10-28). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-08T06:16:23.289Z. First observed: 2026-10-30T06:42:45.896Z. Retrieved: 2026-11-08T06:01:40.612Z. - [GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices](https://arxiv.org/abs/2610.38210) ## Limits The abstract does not identify the five tested models or provide enough experimental detail to assess how broadly the reported gains apply. The reported measurements are attributed to the paper's authors; the supplied evidence is the arXiv abstract, not an independent evaluation. ## Open question How do GAAC's reported compression and serving results vary across the five tested models and the reported device setups? ## Source-stated The authors introduce GAAC as a lossless KV compression and serving system that encodes BF16 exponents and preserves KV values bit-for-bit. > GAAC encodes BF16 exponents using calibrated per-channel base-plus-offset codes over contiguous ranges. - [Supporting source](https://arxiv.org/abs/2610.38210) ## Source-stated Across five tested models, the authors report 1.405–1.407× lossless BF16 KV compression. > Across five tested models, GAAC achieves 1.405-1.407$\\times$ lossless BF16 KV compression. - [Supporting source](https://arxiv.org/abs/2610.38210) ## Source-stated On NVIDIA DGX Spark, the authors report decode-attention and end-to-end W4A16 decode-step speedups. > On NVIDIA DGX Spark, GAAC accelerates decode attention at a 64K-token context by 1.19-1.27$\\times$ and end-to-end W4A16 decode steps by 1.04-1.15$\\times$. - [Supporting source](https://arxiv.org/abs/2610.38210) ## Source-stated For serving, the authors report reduced resident KV memory and higher aggregate throughput under comparable KV-memory budgets. > In serving, it reduces resident KV memory by approximately 26% and increases aggregate throughput by 13-29% under comparable KV-memory budgets. - [Supporting source](https://arxiv.org/abs/2610.38210) ## Escape Meridian interpretation The abstract's reported gains make KV-cache compression a potential way to reduce memory use and speed inference on personal devices; the abstract alone does not establish how broadly those gains apply. > the growing key-value (KV) cache remains a bottleneck for long-context inference on personal devices. - [Supporting source](https://arxiv.org/abs/2610.38210) ## Source and contributor context Guo, Yipin, George, Arun M, Joshi, Siddharth · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Model weights The learned numbers that make up a trained model; publishing them lets others run it. - [Definition reference](https://developers.google.com/machine-learning/glossary#weight) ## Graphics processing unit A chip that performs many calculations at once; the main hardware used to train and run AI models. - [Definition reference](https://en.wikipedia.org/wiki/Graphics_processing_unit) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## NVIDIA NVIDIA develops processors, networking and software used for AI, graphics and scientific computing. - [Explore this context](/entities/nvidia) - [Reference](https://blogs.nvidia.com/) ## Editorial provenance Version 1. Publication ID: 02b3e530-1d35-4756-864e-ca0eed88bcde. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-08T06:16:23.289Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 25c66c49eb2502c1cf92281e9dd39227ffd2d1e78c547361dc7c86488026435e. Source text hash: 82cf4d1495c634172dba883c610045427e054e3c0ba89910c727b35dd661d402. ## Revision and source-check history ### Version 1 — 2026-11-08T06:16:23.289Z Yipin Guo, Arun M George and Siddharth Joshi present GAAC, a system for compressing the key-value (KV) cache used during large language model inference. They argue that this cache can become a memory bottleneck for long-context inference on personal devices, even when model weights have been compressed. GAAC encodes BF16 exponents in groups and reconstructs KV tiles as attention runs; the authors say it preserves KV values bit-for-bit rather than approximating them. Across five tested models, they report 1.405–1.407× lossless BF16 KV compression. On NVIDIA DGX Spark, they report faster decode attention at a 64K-token context and 1.04–1.15× faster end-to-end W4A16 decode steps. In serving, they report about 26% less resident KV memory and 13–29% more aggregate throughput under comparable KV-memory budgets. They also report attention speedups on Apple M4, suggesting the approach is not limited to NVIDIA GPUs in their tests. Escape Meridian interpretation: the results make the cache itself—not only model weights—a potential target for reducing memory use and speeding inference on personal devices. The abstract does not identify the tested models or give enough experimental detail to assess how broadly the reported gains apply. ## Original source arXiv cs.AI · Article Original source date: Oct 28, 2026 (2026-10-28). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-08T06:16:23.289Z. First observed: 2026-10-30T06:42:45.896Z. Retrieved: 2026-11-08T06:01:40.612Z. - [GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices](https://arxiv.org/abs/2610.38210) ## Limits The abstract does not identify the five tested models or provide enough experimental detail to assess how broadly the reported gains apply. The reported measurements are attributed to the paper's authors; the supplied evidence is the arXiv abstract, not an independent evaluation. ## Open question How do GAAC's reported compression and serving results vary across the five tested models and the reported device setups? ## Source-stated The authors introduce GAAC as a lossless KV compression and serving system that encodes BF16 exponents and preserves KV values bit-for-bit. > GAAC encodes BF16 exponents using calibrated per-channel base-plus-offset codes over contiguous ranges. - [Supporting source](https://arxiv.org/abs/2610.38210) ## Source-stated Across five tested models, the authors report 1.405–1.407× lossless BF16 KV compression. > Across five tested models, GAAC achieves 1.405-1.407$\\times$ lossless BF16 KV compression. - [Supporting source](https://arxiv.org/abs/2610.38210) ## Source-stated On NVIDIA DGX Spark, the authors report decode-attention and end-to-end W4A16 decode-step speedups. > On NVIDIA DGX Spark, GAAC accelerates decode attention at a 64K-token context by 1.19-1.27$\\times$ and end-to-end W4A16 decode steps by 1.04-1.15$\\times$. - [Supporting source](https://arxiv.org/abs/2610.38210) ## Source-stated For serving, the authors report reduced resident KV memory and higher aggregate throughput under comparable KV-memory budgets. > In serving, it reduces resident KV memory by approximately 26% and increases aggregate throughput by 13-29% under comparable KV-memory budgets. - [Supporting source](https://arxiv.org/abs/2610.38210) ## Escape Meridian interpretation The abstract's reported gains make KV-cache compression a potential way to reduce memory use and speed inference on personal devices; the abstract alone does not establish how broadly those gains apply. > the growing key-value (KV) cache remains a bottleneck for long-context inference on personal devices. - [Supporting source](https://arxiv.org/abs/2610.38210) ## Source and contributor context Guo, Yipin, George, Arun M, Joshi, Siddharth · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Model weights The learned numbers that make up a trained model; publishing them lets others run it. - [Definition reference](https://developers.google.com/machine-learning/glossary#weight) ## Graphics processing unit A chip that performs many calculations at once; the main hardware used to train and run AI models. - [Definition reference](https://en.wikipedia.org/wiki/Graphics_processing_unit) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## NVIDIA NVIDIA develops processors, networking and software used for AI, graphics and scientific computing. - [Explore this context](/entities/nvidia) - [Reference](https://blogs.nvidia.com/) ## Editorial provenance Version 1. Publication ID: 02b3e530-1d35-4756-864e-ca0eed88bcde. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-08T06:16:23.289Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 25c66c49eb2502c1cf92281e9dd39227ffd2d1e78c547361dc7c86488026435e. Source text hash: 82cf4d1495c634172dba883c610045427e054e3c0ba89910c727b35dd661d402. ### passed — 2026-11-08T06:16:23.289Z — anthropic/claude-opus-5.5 Publication: 02b3e530-1d35-4756-864e-ca0eed88bcde attribution — pass: The draft correctly credits Yipin Guo, Arun M George and Siddharth Joshi as the authors of GAAC. It attributes the bottleneck argument and all measurements to the authors, and it labels the one inference as Escape Meridian interpretation. > Authors:Yipin Guo, Arun M George, Siddharth Joshi summary — pass: The draft leads with the actor and the development, then gives the authors' motivating problem: the KV cache stays a bottleneck even after weights are compressed. It says what lossless means here (bit-for-bit preservation) and reports the figures faithfully. Terms such as BF16, W4A16 and decode attention are kept exactly, consistent with tooltip policy. The draft softens the source's 'demonstrating portability beyond NVIDIA GPUs' to a test-limited claim, which is cautious but not distorting. The question is substantive and non-leading. It names its gaps, the unidentified models and limited experimental detail, without inventing context. > the growing key-value (KV) cache remains a bottleneck for long-context inference on personal devices date — pass: originalPublished is 2026-10-28, which matches the v1 submission date. It does not use the later v2 revision (4 Nov) or the feed date (6 Nov). > \[Submitted on 28 Oct 2026 (v1), last revised 4 Nov 2026 (this version, v2)\] support — pass: Each numeric claim matches the abstract: 1.405-1.407x compression, 1.04-1.15x decode steps, about 26% less resident KV memory, and 13-29% more aggregate throughput. The bit-for-bit claim is stated in the source. The interpretation is conditional ('potential target') and grounded in the authors' stated bottleneck. The draft notes that the evidence is an abstract, not an independent evaluation, and it does not overclaim real-world benefits. > GAAC adapts code widths across small groups while preserving KV values bit-for-bit. media — pass: The selected media ID is the verified direct link to the original arXiv abstract page. It is the most direct way to access the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/e6650ccb6971362e09ba96d30fe4609368eed1ba729fe2190b5f94861a1c1029/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-gaac-lossless-kv-compression-for-bandwidth-e6650ccb.json)