Huang, Wen, Liu, Jack · Nov 10, 2026
How Layer Deadlines Shape LLM Lookup Table Placement
Markdown · JSON · Original source
Wen Huang and Jack Liu argue that where lookup tables reside should depend on when each layer needs its data, as well as session reuse and contention on a shared device. Their system, SlackTier, starts table lookups at the beginning of a step, reads the layer with the shorter deadline first, and splits a small row cache across layers. The authors report that Layer 14’s table can reside on NVMe without a cache and with no measurable decode cost. In their agent and chat replays, placing both tables on four NVMe disks raises mean per output token by at most 1.4% over unmodified SGLang, while reducing host memory use from 212 to 34 GiB. They also report that RDMA reads from another node’s memory allow both tables to run without a cache. For a KV-constrained agent workload, they use the freed memory as host KV cache and report 36% higher median throughput and a reduction in median p90 time to first token from 38 to 24 seconds. These results connect table placement to a practical tradeoff: memory freed from lookup tables can support KV cache, but the reported performance depends on the workloads and measured read times.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 10, 2026 (2026-11-10).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-12T01:05:06.982Z. First observed: 2026-11-11T05:09:22.170Z. Retrieved: 2026-11-12T01:03:48.741Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The supplied evidence gives the paper’s abstract but not its methods or detailed results; the reported performance should therefore be read in the scope of the workloads and measurements described there.
Open question
How do the measured read times and workload conditions affect whether SlackTier’s reported results hold beyond the agent and chat replays described?
Source-stated
Huang and Liu argue that table placement depends on per-layer deadlines, session reuse, and contention on a shared device.
We show that this deadline, together with session reuse and contention on a shared device, decides which tier each table can use and which layers need cache
Source-stated
The authors describe SlackTier as issuing lookups at step start, prioritizing the short-window layer, and splitting a small GPU row cache across layers.
It reads the short-window layer first and splits a small GPU row cache across layers.
Source-stated
The authors report that Layer 14’s table can be placed on NVMe without a cache and with no measurable decode cost.
Layer 14's table can go to NVMe uncached with no measurable decode cost.
Source-stated
In agent and chat replays, the authors report at most a 1.4% increase in mean time per output token with both tables on four NVMe disks, while host memory falls from 212 to 34 GiB.
With both tables on four NVMe disks, mean time per output token on agent and chat replays rises at most 1.4% over unmodified SGLang, and host memory falls from 212 to 34 GiB.
Source-stated
For a KV-constrained agent workload, the authors report improved median throughput and lower median p90 time to first token when freed memory is used as host KV cache.
using the freed memory as host KV cache at the same memory budget raises median throughput by 36% and cuts median p90 time to first token from 38 to 24 s.
Source and contributor context
Huang, Wen, Liu, Jack · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Graphics processing unit
A chip that performs many calculations at once; the main hardware used to train and run AI models.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: e5ae0632-7c46-4ce6-8a6f-97d72250d2a6.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T01:05:06.982Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: b835d3bd2c111d3deb7ef3dea0bbef3c32c62a3c4e88127e7be7924c5f93e9c2. Source text hash: 754d68d83ab12368ecf8aed1f10e3ad484bea422733eb8c746fe3c6d49df4116.
Revision and source-check history
Version 1 · 2026-11-12T01:05:06.982Z
How Layer Deadlines Shape LLM Lookup Table Placement
Wen Huang and Jack Liu argue that where lookup tables reside should depend on when each layer needs its data, as well as session reuse and contention on a shared device. Their system, SlackTier, starts table lookups at the beginning of a step, reads the layer with the shorter deadline first, and splits a small row cache across layers. The authors report that Layer 14’s table can reside on NVMe without a cache and with no measurable decode cost. In their agent and chat replays, placing both tables on four NVMe disks raises mean per output token by at most 1.4% over unmodified SGLang, while reducing host memory use from 212 to 34 GiB. They also report that RDMA reads from another node’s memory allow both tables to run without a cache. For a KV-constrained agent workload, they use the freed memory as host KV cache and report 36% higher median throughput and a reduction in median p90 time to first token from 38 to 24 seconds. These results connect table placement to a practical tradeoff: memory freed from lookup tables can support KV cache, but the reported performance depends on the workloads and measured read times.
Original source
cs.AI · Article
Source details
Original source date: Nov 10, 2026 (2026-11-10).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-12T01:05:06.982Z. First observed: 2026-11-11T05:09:22.170Z. Retrieved: 2026-11-12T01:03:48.741Z.
Limits
The supplied evidence gives the paper’s abstract but not its methods or detailed results; the reported performance should therefore be read in the scope of the workloads and measurements described there.
Open question
How do the measured read times and workload conditions affect whether SlackTier’s reported results hold beyond the agent and chat replays described?
Source-stated
Huang and Liu argue that table placement depends on per-layer deadlines, session reuse, and contention on a shared device.
We show that this deadline, together with session reuse and contention on a shared device, decides which tier each table can use and which layers need cache
Source-stated
The authors describe SlackTier as issuing lookups at step start, prioritizing the short-window layer, and splitting a small GPU row cache across layers.
It reads the short-window layer first and splits a small GPU row cache across layers.
Source-stated
The authors report that Layer 14’s table can be placed on NVMe without a cache and with no measurable decode cost.
Layer 14's table can go to NVMe uncached with no measurable decode cost.
Source-stated
In agent and chat replays, the authors report at most a 1.4% increase in mean time per output token with both tables on four NVMe disks, while host memory falls from 212 to 34 GiB.
With both tables on four NVMe disks, mean time per output token on agent and chat replays rises at most 1.4% over unmodified SGLang, and host memory falls from 212 to 34 GiB.
Source-stated
For a KV-constrained agent workload, the authors report improved median throughput and lower median p90 time to first token when freed memory is used as host KV cache.
using the freed memory as host KV cache at the same memory budget raises median throughput by 36% and cuts median p90 time to first token from 38 to 24 s.
Source and contributor context
Huang, Wen, Liu, Jack · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Graphics processing unit
A chip that performs many calculations at once; the main hardware used to train and run AI models.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: e5ae0632-7c46-4ce6-8a6f-97d72250d2a6.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T01:05:06.982Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: b835d3bd2c111d3deb7ef3dea0bbef3c32c62a3c4e88127e7be7924c5f93e9c2. Source text hash: 754d68d83ab12368ecf8aed1f10e3ad484bea422733eb8c746fe3c6d49df4116.
uncertain · 2026-11-12T01:05:06.982Z · anthropic/claude-opus-5.5
Publication: e5ae0632-7c46-4ce6-8a6f-97d72250d2a6
attribution — pass: The reading attributes the argument and SlackTier results to authors Wen Huang and Jack Liu and keeps the results framed as their reported findings.
Authors:Wen Huang, Jack Liu
summary — uncertain: The figures match the abstract, but a general reader lacks essential context. The draft never says the tables are DeepSeek-V4.1's 203 GB Engram n-gram tables, which currently occupy most host DRAM. It also leaves central terms unexplained: lookup tables, NVMe, RDMA, KV cache, SGLang, decode, and p90 time to first token. RDMA is expanded in the source. The draft omits that one RDMA copy serves two instances. Smallest repair: add the DeepSeek-V4.1/Engram context and the 190-of-223 GiB memory problem, then use term tooltips for the jargon.
DeepSeek-V4.1's 203 GB of Engram n-gram tables sit in host DRAM today, taking 190 of the 223 GiB a 4-GPU serving process uses.
date — pass: originalPublished 2026-11-10 matches the arXiv v1 submission date. It is not the later feed date.
[Submitted on 10 Nov 2026]
support — pass: Each claim matches a verbatim abstract excerpt. The closing tradeoff stays within the source's scope: freed memory used as host KV cache, and a slowdown model based on measured read times. The question is open, not leading.
we give a model that predicts the resulting slowdown from measured read times
media — pass: The selected media is the verified direct arXiv abstract page for this paper, the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A language model with many learned numerical settings, used to process and generate language.
Definition referenceA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referenceA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv