Zheng, Bowen, Luo, Weijian, Yang, Guang, Zhang, Colin, Hu, Tianyang · May 7, 2026
Taming the Entropy Cliff: Variable Codebook Size Quantization for Autoregressive Visual Generation
Markdown · JSON · Original source
The paper’s authors argue that discrete visual can use their finite-data entropy budget too quickly when every sequence position shares the same codebook. They call the resulting rapid drop in per-position empirical conditional entropy the “Entropy Cliff”: on ImageNet with K=16384, they report that after about 2 of 256 positions the conditional distribution is essentially deterministic, as most prefixes already identify individual dataset images. Their proposed Variable Codebook Size (VCQ) grows codebook size along while leaving the loss function, and autoregressive training procedure unchanged. With a vanilla autoregressive Transformer and standard next-token prediction, VCQ-Base reports gFID w/o CFG of 14.80 on ImageNet 256×256, versus 27.98 and 18.16 for constant codebooks of size 16384 and 8192. The authors also report gFID 1.71 at 684M autoregressive parameters, without semantic regularization or causal alignment. A linear probe using only the first 10 positions reaches 43.8% top-1 accuracy, compared with 27.1% for uniform codebooks. Their central argument is that codebook capacity’s distribution across positions matters, not just its total. The abstract reports these results on ImageNet; how VCQ performs on other datasets remains an open test.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: May 7, 2026 (2026-05-07).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-08T19:23:33.642Z. First observed: 2026-11-06T05:01:00.407Z. Retrieved: 2026-11-08T19:16:53.718Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The reported results in the abstract are on ImageNet; it provides no results on other datasets. The evidence is the paper’s abstract, not its full methods or results.
Open question
Do the reported VCQ results hold on datasets beyond ImageNet?
Source-stated
The authors report that with a constant codebook of size 16384 on ImageNet, the conditional distribution becomes essentially deterministic after about 2 of 256 positions, which they call the Entropy Cliff.
On ImageNet with $K=16384$, this happens after only about 2 out of 256 positions, when most prefixes already identify individual images in the dataset. We call this phenomenon the Entropy Cliff.
Source-stated
The authors propose VCQ, which increases codebook size along the sequence while leaving the loss function, parameter count, and autoregressive training procedure unchanged.
the codebook size $K_t$ grows monotonically along the sequence from $K_{\min}=2$ to $K_{\max}$, leaving the loss function, parameter count, and AR training procedure unchanged.
Source-stated
On ImageNet 256×256, the authors report gFID w/o CFG of 14.80 for VCQ-Base, compared with 27.98 and 18.16 for constant codebooks of size 16384 and 8192.
VCQ-Base achieves gFID w/o CFG 14.80 on ImageNet $256\times256$, compared with 27.98 and 18.16 for constant codebooks of size 16384 and 8192, respectively.
Source-stated
The authors report that a linear probe using features from the first 10 positions reaches 43.8% top-1 accuracy on ImageNet, versus 27.1% for uniform codebooks.
a linear probe on features from only the first 10 positions reaches 43.8% top-1 accuracy on ImageNet, compared to 27.1% for uniform codebooks.
Escape Meridian interpretation
The abstract reports ImageNet results; testing whether the reported results extend to other datasets would establish the scope of the findings.
On ImageNet with $K=16384$, this happens after only about 2 out of 256 positions
Source and contributor context
Zheng, Bowen, Luo, Weijian, Yang, Guang, Zhang, Colin, Hu, Tianyang · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Quantization
Storing a model's numbers at lower precision so it uses less memory and runs faster, sometimes at a cost in accuracy.
Model tokens
Small chunks of text, often parts of words, that a language model reads and writes; usage and prices are often counted in tokens.
Model parameters
The internal numbers a model adjusts during training; their count is a rough measure of a model's size.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
The Sequence
The Sequence publishes machine-learning analysis and interviews whose guest answers retain their own attribution.
Editorial provenance
Version 1. Publication ID: 300462da-882e-4136-853a-aea259cdd127.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-08T19:23:33.642Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 1e98aeb0d799c73534efb56cf4e4f00a432b05bbfad3e155e96b3cf835ffd957. Source text hash: 21f23c06b8d9e65224b4fd98ba26a192ae4c489249d5efaacccfa6ffb39ec93a.
Revision and source-check history
Version 1 · 2026-11-08T19:23:33.642Z
Taming the Entropy Cliff: Variable Codebook Size Quantization for Autoregressive Visual Generation
The paper’s authors argue that discrete visual can use their finite-data entropy budget too quickly when every sequence position shares the same codebook. They call the resulting rapid drop in per-position empirical conditional entropy the “Entropy Cliff”: on ImageNet with K=16384, they report that after about 2 of 256 positions the conditional distribution is essentially deterministic, as most prefixes already identify individual dataset images. Their proposed Variable Codebook Size (VCQ) grows codebook size along while leaving the loss function, and autoregressive training procedure unchanged. With a vanilla autoregressive Transformer and standard next-token prediction, VCQ-Base reports gFID w/o CFG of 14.80 on ImageNet 256×256, versus 27.98 and 18.16 for constant codebooks of size 16384 and 8192. The authors also report gFID 1.71 at 684M autoregressive parameters, without semantic regularization or causal alignment. A linear probe using only the first 10 positions reaches 43.8% top-1 accuracy, compared with 27.1% for uniform codebooks. Their central argument is that codebook capacity’s distribution across positions matters, not just its total. The abstract reports these results on ImageNet; how VCQ performs on other datasets remains an open test.
Original source
cs.AI · Article
Source details
Original source date: May 7, 2026 (2026-05-07).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-08T19:23:33.642Z. First observed: 2026-11-06T05:01:00.407Z. Retrieved: 2026-11-08T19:16:53.718Z.
Limits
The reported results in the abstract are on ImageNet; it provides no results on other datasets. The evidence is the paper’s abstract, not its full methods or results.
Open question
Do the reported VCQ results hold on datasets beyond ImageNet?
Source-stated
The authors report that with a constant codebook of size 16384 on ImageNet, the conditional distribution becomes essentially deterministic after about 2 of 256 positions, which they call the Entropy Cliff.
On ImageNet with $K=16384$, this happens after only about 2 out of 256 positions, when most prefixes already identify individual images in the dataset. We call this phenomenon the Entropy Cliff.
Source-stated
The authors propose VCQ, which increases codebook size along the sequence while leaving the loss function, parameter count, and autoregressive training procedure unchanged.
the codebook size $K_t$ grows monotonically along the sequence from $K_{\min}=2$ to $K_{\max}$, leaving the loss function, parameter count, and AR training procedure unchanged.
Source-stated
On ImageNet 256×256, the authors report gFID w/o CFG of 14.80 for VCQ-Base, compared with 27.98 and 18.16 for constant codebooks of size 16384 and 8192.
VCQ-Base achieves gFID w/o CFG 14.80 on ImageNet $256\times256$, compared with 27.98 and 18.16 for constant codebooks of size 16384 and 8192, respectively.
Source-stated
The authors report that a linear probe using features from the first 10 positions reaches 43.8% top-1 accuracy on ImageNet, versus 27.1% for uniform codebooks.
a linear probe on features from only the first 10 positions reaches 43.8% top-1 accuracy on ImageNet, compared to 27.1% for uniform codebooks.
Escape Meridian interpretation
The abstract reports ImageNet results; testing whether the reported results extend to other datasets would establish the scope of the findings.
On ImageNet with $K=16384$, this happens after only about 2 out of 256 positions
Source and contributor context
Zheng, Bowen, Luo, Weijian, Yang, Guang, Zhang, Colin, Hu, Tianyang · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Quantization
Storing a model's numbers at lower precision so it uses less memory and runs faster, sometimes at a cost in accuracy.
Model tokens
Small chunks of text, often parts of words, that a language model reads and writes; usage and prices are often counted in tokens.
Model parameters
The internal numbers a model adjusts during training; their count is a rough measure of a model's size.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
The Sequence
The Sequence publishes machine-learning analysis and interviews whose guest answers retain their own attribution.
Editorial provenance
Version 1. Publication ID: 300462da-882e-4136-853a-aea259cdd127.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-08T19:23:33.642Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 1e98aeb0d799c73534efb56cf4e4f00a432b05bbfad3e155e96b3cf835ffd957. Source text hash: 21f23c06b8d9e65224b4fd98ba26a192ae4c489249d5efaacccfa6ffb39ec93a.
uncertain · 2026-11-08T19:23:33.642Z · anthropic/claude-opus-5.5
Publication: 300462da-882e-4136-853a-aea259cdd127
attribution — pass: The draft consistently attributes the Entropy Cliff observation, VCQ proposal and benchmark numbers to the paper's authors. It uses 'they report' and 'the authors also report' rather than presenting the claims as established fact.
Authors:Bowen Zheng, Weijian Luo, Guang Yang, Colin Zhang, Tianyang Hu
summary — uncertain: The summary closely tracks the abstract but leaves central jargon unexplained for a general reader. It never says what a discrete visual tokenizer or codebook is, what gFID measures or that lower is better, what 'w/o CFG' or a linear probe means, or what the authors mean by conditional entropy becoming deterministic. Without this, a reader cannot tell that 14.80 beats 27.98 or why the result matters. Smallest repair: add brief plain-language glosses or term-library tooltips for codebook, tokenizer, gFID (lower is better) and linear probe, keeping the original terms. The title and question are faithful.
Most discrete visual tokenizers rely on a default design: every position in the sequence shares the same codebook.
date — pass: originalPublished 2026-05-07 matches the v1 submission date. The 5 Nov 2026 revision and the November feed date are correctly not used.
[Submitted on 7 May 2026 (v1), last revised 5 Nov 2026 (this version, v2)]
support — uncertain: The numeric claims match the abstract exactly. However, the limitation that the abstract 'provides no results on other datasets' and the statement that 'the abstract reports these results on ImageNet' omit the abstract's language comparison. In that comparison the authors say entropy declines more gradually, and they use this contrast to support their argument. Repair: say that VCQ's generation results are reported only on ImageNet, and mention the language comparison. The conditional question about generalization to other datasets is otherwise reasonable.
The empirical entropy decreases more gradually in our language comparison, where the effective entropy per position stays below the codebook capacity.
media — pass: The selected media is the verified arXiv abstract page for this paper, which is the direct original source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Small chunks of text, often parts of words, that a language model reads and writes; usage and prices are often counted in tokens.
Definition referenceStoring a model's numbers at lower precision so it uses less memory and runs faster, sometimes at a cost in accuracy.
Definition referenceThe Sequence publishes machine-learning analysis and interviews whose guest answers retain their own attribution.
Explore The SequenceThe internal numbers a model adjusts during training; their count is a rough measure of a model's size.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivSmall chunks of text, often parts of words, that a language model reads and writes; usage and prices are often counted in tokens.
Definition referenceStoring a model's numbers at lower precision so it uses less memory and runs faster, sometimes at a cost in accuracy.
Definition referenceThe Sequence publishes machine-learning analysis and interviews whose guest answers retain their own attribution.
Explore The SequenceThe internal numbers a model adjusts during training; their count is a rough measure of a model's size.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv