{"id":"74858ea30a67436d0b488bc23427cedd337835cf22c50ad85841b00686cc1e76","title":"Taming the Entropy Cliff: Variable Codebook Size Quantization for Autoregressive Visual Generation","argument":"The paper’s authors argue that discrete visual tokenizers can use their finite-data entropy budget too quickly when every sequence position shares the same codebook. They call the resulting rapid drop in per-position empirical conditional entropy the “Entropy Cliff”: on ImageNet with K=16384, they report that after about 2 of 256 positions the conditional distribution is essentially deterministic, as most prefixes already identify individual dataset images. Their proposed Variable Codebook Size Quantization (VCQ) grows codebook size along the sequence while leaving the loss function, parameter count and autoregressive training procedure unchanged. With a vanilla autoregressive Transformer and standard next-token prediction, VCQ-Base reports gFID w/o CFG of 14.80 on ImageNet 256×256, versus 27.98 and 18.16 for constant codebooks of size 16384 and 8192. The authors also report gFID 1.71 at 684M autoregressive parameters, without semantic regularization or causal alignment. A linear probe using only the first 10 positions reaches 43.8% top-1 accuracy, compared with 27.1% for uniform codebooks. Their central argument is that codebook capacity’s distribution across positions matters, not just its total. The abstract reports these results on ImageNet; how VCQ performs on other datasets remains an open test.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported results in the abstract are on ImageNet; it provides no results on other datasets. The evidence is the paper’s abstract, not its full methods or results.","speaker":{"name":"Zheng, Bowen, Luo, Weijian, Yang, Guang, Zhang, Colin, Hu, Tianyang","roleAtSource":"Article author"},"source":{"title":"Taming the Entropy Cliff: Variable Codebook Size Quantization for Autoregressive Visual Generation","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2605.06207","publishedAt":"2026-05-07","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors report that with a constant codebook of size 16384 on ImageNet, the conditional distribution becomes essentially deterministic after about 2 of 256 positions, which they call the Entropy Cliff.","excerpt":"On ImageNet with $K=16384$, this happens after only about 2 out of 256 positions, when most prefixes already identify individual images in the dataset. We call this phenomenon the Entropy Cliff.","url":"https://arxiv.org/abs/2605.06207","relationship":"Source-stated"},{"text":"The authors propose VCQ, which increases codebook size along the sequence while leaving the loss function, parameter count, and autoregressive training procedure unchanged.","excerpt":"the codebook size $K_t$ grows monotonically along the sequence from $K_{\\min}=2$ to $K_{\\max}$, leaving the loss function, parameter count, and AR training procedure unchanged.","url":"https://arxiv.org/abs/2605.06207","relationship":"Source-stated"},{"text":"On ImageNet 256×256, the authors report gFID w/o CFG of 14.80 for VCQ-Base, compared with 27.98 and 18.16 for constant codebooks of size 16384 and 8192.","excerpt":"VCQ-Base achieves gFID w/o CFG 14.80 on ImageNet $256\\times256$, compared with 27.98 and 18.16 for constant codebooks of size 16384 and 8192, respectively.","url":"https://arxiv.org/abs/2605.06207","relationship":"Source-stated"},{"text":"The authors report that a linear probe using features from the first 10 positions reaches 43.8% top-1 accuracy on ImageNet, versus 27.1% for uniform codebooks.","excerpt":"a linear probe on features from only the first 10 positions reaches 43.8% top-1 accuracy on ImageNet, compared to 27.1% for uniform codebooks.","url":"https://arxiv.org/abs/2605.06207","relationship":"Source-stated"},{"text":"The abstract reports ImageNet results; testing whether the reported results extend to other datasets would establish the scope of the findings.","excerpt":"On ImageNet with $K=16384$, this happens after only about 2 out of 256 positions","url":"https://arxiv.org/abs/2605.06207","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2605.06207","checkedAt":"2026-11-08T20:26:08.275Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Do the reported VCQ results hold on datasets beyond ImageNet?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Taming the Entropy Cliff: Variable Codebook Size Quantization for Autoregressive Visual Generation","summary":"The paper’s authors argue that discrete visual tokenizers can use their finite-data entropy budget too quickly when every sequence position shares the same codebook. They call the resulting rapid drop in per-position empirical conditional entropy the “Entropy Cliff”: on ImageNet with K=16384, they report that after about 2 of 256 positions the conditional distribution is essentially deterministic, as most prefixes already identify individual dataset images. Their proposed Variable Codebook Size Quantization (VCQ) grows codebook size along the sequence while leaving the loss function, parameter count and autoregressive training procedure unchanged. With a vanilla autoregressive Transformer and standard next-token prediction, VCQ-Base reports gFID w/o CFG of 14.80 on ImageNet 256×256, versus 27.98 and 18.16 for constant codebooks of size 16384 and 8192. The authors also report gFID 1.71 at 684M autoregressive parameters, without semantic regularization or causal alignment. A linear probe using only the first 10 positions reaches 43.8% top-1 accuracy, compared with 27.1% for uniform codebooks. Their central argument is that codebook capacity’s distribution across positions matters, not just its total. The abstract reports these results on ImageNet; how VCQ performs on other datasets remains an open test.","question":"Do the reported VCQ results hold on datasets beyond ImageNet?","claims":[{"text":"The authors report that with a constant codebook of size 16384 on ImageNet, the conditional distribution becomes essentially deterministic after about 2 of 256 positions, which they call the Entropy Cliff.","excerpt":"On ImageNet with $K=16384$, this happens after only about 2 out of 256 positions, when most prefixes already identify individual images in the dataset. We call this phenomenon the Entropy Cliff.","support":"source-stated"},{"text":"The authors propose VCQ, which increases codebook size along the sequence while leaving the loss function, parameter count, and autoregressive training procedure unchanged.","excerpt":"the codebook size $K_t$ grows monotonically along the sequence from $K_{\\min}=2$ to $K_{\\max}$, leaving the loss function, parameter count, and AR training procedure unchanged.","support":"source-stated"},{"text":"On ImageNet 256×256, the authors report gFID w/o CFG of 14.80 for VCQ-Base, compared with 27.98 and 18.16 for constant codebooks of size 16384 and 8192.","excerpt":"VCQ-Base achieves gFID w/o CFG 14.80 on ImageNet $256\\times256$, compared with 27.98 and 18.16 for constant codebooks of size 16384 and 8192, respectively.","support":"source-stated"},{"text":"The authors report that a linear probe using features from the first 10 positions reaches 43.8% top-1 accuracy on ImageNet, versus 27.1% for uniform codebooks.","excerpt":"a linear probe on features from only the first 10 positions reaches 43.8% top-1 accuracy on ImageNet, compared to 27.1% for uniform codebooks.","support":"source-stated"},{"text":"The abstract reports ImageNet results; testing whether the reported results extend to other datasets would establish the scope of the findings.","excerpt":"On ImageNet with $K=16384$, this happens after only about 2 out of 256 positions","support":"interpretation"}],"limitations":["The reported results in the abstract are on ImageNet; it provides no results on other datasets.","The evidence is the paper’s abstract, not its full methods or results."],"mediaId":"3cf45f06dacdca1602ca1d444001d6ddc88e752154a293b3d7dc27d5920460ce","correctionReason":null,"id":"74858ea30a67436d0b488bc23427cedd337835cf22c50ad85841b00686cc1e76","publicationId":"300462da-882e-4136-853a-aea259cdd127","version":1,"sourceUrl":"https://arxiv.org/abs/2605.06207","sourceTitle":"Taming the Entropy Cliff: Variable Codebook Size Quantization for Autoregressive Visual Generation","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-05-07","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:01:00.407Z","createdAt":"2026-11-08T19:23:33.642Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"3cf45f06dacdca1602ca1d444001d6ddc88e752154a293b3d7dc27d5920460ce","kind":"article","url":"https://arxiv.org/abs/2605.06207","sourceUrl":"https://arxiv.org/abs/2605.06207","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-08T19:16:53.823Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2605.06207","checkedAt":"2026-11-08T20:26:08.275Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"3cf45f06dacdca1602ca1d444001d6ddc88e752154a293b3d7dc27d5920460ce","confidence":0.93,"provider":"typesafe","model":"jev-1.13.0","inputHash":"41ee01f5cb29842afa4a60abb26983291006b4305066eaee2449b82cb7c2d63f","decidedAt":"2026-11-08T19:16:54.522Z"},"provenance":{"textHash":"21f23c06b8d9e65224b4fd98ba26a192ae4c489249d5efaacccfa6ffb39ec93a","observationId":"1e98aeb0d799c73534efb56cf4e4f00a432b05bbfad3e155e96b3cf835ffd957","retrievedAt":"2026-11-08T19:16:53.718Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"0b4a3615-aba5-4dd5-8a8d-e4df5d8fe730"},"reviewer":{"model":"gpt-6-luna","runId":"c0fc41ea-5cd6-4208-808b-7a8010256f55"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-08T19:23:33.642Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Zheng, Bowen","url":null,"kind":"unknown"},{"name":"Luo, Weijian","url":null,"kind":"unknown"},{"name":"Yang, Guang","url":null,"kind":"unknown"},{"name":"Zhang, Colin","url":null,"kind":"unknown"},{"name":"Hu, Tianyang","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Zheng, Bowen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zheng, Bowen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.06207","locator":"head/meta[name=author|citation_author]"},"assertionId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:26:08.275Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88","sourceUrl":"https://arxiv.org/abs/2605.06207","contentSha256":"560a6913883d5b2cc3bd148f91cf6686e2457fb25e585ef822fa558a78a3a87f","textHash":"21f23c06b8d9e65224b4fd98ba26a192ae4c489249d5efaacccfa6ffb39ec93a","observedAt":"2026-11-08T20:26:08.275Z"}},{"actor":{"name":"Luo, Weijian","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Luo, Weijian","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.06207","locator":"head/meta[name=author|citation_author]"},"assertionId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:26:08.275Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88","sourceUrl":"https://arxiv.org/abs/2605.06207","contentSha256":"560a6913883d5b2cc3bd148f91cf6686e2457fb25e585ef822fa558a78a3a87f","textHash":"21f23c06b8d9e65224b4fd98ba26a192ae4c489249d5efaacccfa6ffb39ec93a","observedAt":"2026-11-08T20:26:08.275Z"}},{"actor":{"name":"Yang, Guang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yang, Guang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.06207","locator":"head/meta[name=author|citation_author]"},"assertionId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:26:08.275Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88","sourceUrl":"https://arxiv.org/abs/2605.06207","contentSha256":"560a6913883d5b2cc3bd148f91cf6686e2457fb25e585ef822fa558a78a3a87f","textHash":"21f23c06b8d9e65224b4fd98ba26a192ae4c489249d5efaacccfa6ffb39ec93a","observedAt":"2026-11-08T20:26:08.275Z"}},{"actor":{"name":"Zhang, Colin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Colin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.06207","locator":"head/meta[name=author|citation_author]"},"assertionId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:26:08.275Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88","sourceUrl":"https://arxiv.org/abs/2605.06207","contentSha256":"560a6913883d5b2cc3bd148f91cf6686e2457fb25e585ef822fa558a78a3a87f","textHash":"21f23c06b8d9e65224b4fd98ba26a192ae4c489249d5efaacccfa6ffb39ec93a","observedAt":"2026-11-08T20:26:08.275Z"}},{"actor":{"name":"Hu, Tianyang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Hu, Tianyang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.06207","locator":"head/meta[name=author|citation_author]"},"assertionId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:26:08.275Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88","sourceUrl":"https://arxiv.org/abs/2605.06207","contentSha256":"560a6913883d5b2cc3bd148f91cf6686e2457fb25e585ef822fa558a78a3a87f","textHash":"21f23c06b8d9e65224b4fd98ba26a192ae4c489249d5efaacccfa6ffb39ec93a","observedAt":"2026-11-08T20:26:08.275Z"}}],"participationCoverage":{"total":5,"returned":5,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2605.06207","snapshotId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88","contentSha256":"560a6913883d5b2cc3bd148f91cf6686e2457fb25e585ef822fa558a78a3a87f","textHash":"21f23c06b8d9e65224b4fd98ba26a192ae4c489249d5efaacccfa6ffb39ec93a","observedAt":"2026-11-08T20:26:08.275Z","basis":"publisher-page-metadata","sourceLinks":47,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/74858ea30a67436d0b488bc23427cedd337835cf22c50ad85841b00686cc1e76/source"}},"canonicalUrl":"https://escapemeridian.com/readings/taming-the-entropy-cliff-variable-codebook-size-quantization-74858ea3","formats":{"html":"/readings/taming-the-entropy-cliff-variable-codebook-size-quantization-74858ea3","markdown":"/readings/taming-the-entropy-cliff-variable-codebook-size-quantization-74858ea3.md","json":"/readings/taming-the-entropy-cliff-variable-codebook-size-quantization-74858ea3.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-quantization","name":"Quantization","explanation":"Storing a model's numbers at lower precision so it uses less memory and runs faster, sometimes at a cost in accuracy.","referenceUrl":"https://developers.google.com/machine-learning/glossary#quantization"},{"id":"term-tokens","name":"Model tokens","explanation":"Small chunks of text, often parts of words, that a language model reads and writes; usage and prices are often counted in tokens.","referenceUrl":"https://developers.google.com/machine-learning/glossary#token"},{"id":"term-parameters","name":"Model parameters","explanation":"The internal numbers a model adjusts during training; their count is a rough measure of a model's size.","referenceUrl":"https://developers.google.com/machine-learning/glossary#parameter"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"},{"id":"the-sequence","name":"The Sequence","kind":"organization","context":"The Sequence publishes machine-learning analysis and interviews whose guest answers retain their own attribution.","referenceUrls":["https://thesequence.substack.com/","https://thesequence.substack.com/p/the-sequence-chat-emad-mostaque-stability"],"path":"/entities/the-sequence","json":"/entities/the-sequence.json","markdown":"/entities/the-sequence.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"74858ea30a67436d0b488bc23427cedd337835cf22c50ad85841b00686cc1e76","revision":"300462da-882e-4136-853a-aea259cdd127","revisionKind":"publication-id","title":"Taming the Entropy Cliff: Variable Codebook Size Quantization for Autoregressive Visual Generation","url":"https://escapemeridian.com/readings/taming-the-entropy-cliff-variable-codebook-size-quantization-74858ea3"},"source":{"type":"source","id":"https://arxiv.org/abs/2605.06207","revision":"21f23c06b8d9e65224b4fd98ba26a192ae4c489249d5efaacccfa6ffb39ec93a","revisionKind":"sha256-source-text","title":"Taming the Entropy Cliff: Variable Codebook Size Quantization for Autoregressive Visual Generation","url":"https://arxiv.org/abs/2605.06207"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"74858ea30a67436d0b488bc23427cedd337835cf22c50ad85841b00686cc1e76","title":"Taming the Entropy Cliff: Variable Codebook Size Quantization for Autoregressive Visual Generation","argument":"The paper’s authors argue that discrete visual tokenizers can use their finite-data entropy budget too quickly when every sequence position shares the same codebook. They call the resulting rapid drop in per-position empirical conditional entropy the “Entropy Cliff”: on ImageNet with K=16384, they report that after about 2 of 256 positions the conditional distribution is essentially deterministic, as most prefixes already identify individual dataset images. Their proposed Variable Codebook Size Quantization (VCQ) grows codebook size along the sequence while leaving the loss function, parameter count and autoregressive training procedure unchanged. With a vanilla autoregressive Transformer and standard next-token prediction, VCQ-Base reports gFID w/o CFG of 14.80 on ImageNet 256×256, versus 27.98 and 18.16 for constant codebooks of size 16384 and 8192. The authors also report gFID 1.71 at 684M autoregressive parameters, without semantic regularization or causal alignment. A linear probe using only the first 10 positions reaches 43.8% top-1 accuracy, compared with 27.1% for uniform codebooks. Their central argument is that codebook capacity’s distribution across positions matters, not just its total. The abstract reports these results on ImageNet; how VCQ performs on other datasets remains an open test.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported results in the abstract are on ImageNet; it provides no results on other datasets. The evidence is the paper’s abstract, not its full methods or results.","speaker":{"name":"Zheng, Bowen, Luo, Weijian, Yang, Guang, Zhang, Colin, Hu, Tianyang","roleAtSource":"Article author"},"source":{"title":"Taming the Entropy Cliff: Variable Codebook Size Quantization for Autoregressive Visual Generation","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2605.06207","publishedAt":"2026-05-07","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors report that with a constant codebook of size 16384 on ImageNet, the conditional distribution becomes essentially deterministic after about 2 of 256 positions, which they call the Entropy Cliff.","excerpt":"On ImageNet with $K=16384$, this happens after only about 2 out of 256 positions, when most prefixes already identify individual images in the dataset. We call this phenomenon the Entropy Cliff.","url":"https://arxiv.org/abs/2605.06207","relationship":"Source-stated"},{"text":"The authors propose VCQ, which increases codebook size along the sequence while leaving the loss function, parameter count, and autoregressive training procedure unchanged.","excerpt":"the codebook size $K_t$ grows monotonically along the sequence from $K_{\\min}=2$ to $K_{\\max}$, leaving the loss function, parameter count, and AR training procedure unchanged.","url":"https://arxiv.org/abs/2605.06207","relationship":"Source-stated"},{"text":"On ImageNet 256×256, the authors report gFID w/o CFG of 14.80 for VCQ-Base, compared with 27.98 and 18.16 for constant codebooks of size 16384 and 8192.","excerpt":"VCQ-Base achieves gFID w/o CFG 14.80 on ImageNet $256\\times256$, compared with 27.98 and 18.16 for constant codebooks of size 16384 and 8192, respectively.","url":"https://arxiv.org/abs/2605.06207","relationship":"Source-stated"},{"text":"The authors report that a linear probe using features from the first 10 positions reaches 43.8% top-1 accuracy on ImageNet, versus 27.1% for uniform codebooks.","excerpt":"a linear probe on features from only the first 10 positions reaches 43.8% top-1 accuracy on ImageNet, compared to 27.1% for uniform codebooks.","url":"https://arxiv.org/abs/2605.06207","relationship":"Source-stated"},{"text":"The abstract reports ImageNet results; testing whether the reported results extend to other datasets would establish the scope of the findings.","excerpt":"On ImageNet with $K=16384$, this happens after only about 2 out of 256 positions","url":"https://arxiv.org/abs/2605.06207","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2605.06207","checkedAt":"2026-11-08T20:26:08.275Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Do the reported VCQ results hold on datasets beyond ImageNet?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Taming the Entropy Cliff: Variable Codebook Size Quantization for Autoregressive Visual Generation","summary":"The paper’s authors argue that discrete visual tokenizers can use their finite-data entropy budget too quickly when every sequence position shares the same codebook. They call the resulting rapid drop in per-position empirical conditional entropy the “Entropy Cliff”: on ImageNet with K=16384, they report that after about 2 of 256 positions the conditional distribution is essentially deterministic, as most prefixes already identify individual dataset images. Their proposed Variable Codebook Size Quantization (VCQ) grows codebook size along the sequence while leaving the loss function, parameter count and autoregressive training procedure unchanged. With a vanilla autoregressive Transformer and standard next-token prediction, VCQ-Base reports gFID w/o CFG of 14.80 on ImageNet 256×256, versus 27.98 and 18.16 for constant codebooks of size 16384 and 8192. The authors also report gFID 1.71 at 684M autoregressive parameters, without semantic regularization or causal alignment. A linear probe using only the first 10 positions reaches 43.8% top-1 accuracy, compared with 27.1% for uniform codebooks. Their central argument is that codebook capacity’s distribution across positions matters, not just its total. The abstract reports these results on ImageNet; how VCQ performs on other datasets remains an open test.","question":"Do the reported VCQ results hold on datasets beyond ImageNet?","claims":[{"text":"The authors report that with a constant codebook of size 16384 on ImageNet, the conditional distribution becomes essentially deterministic after about 2 of 256 positions, which they call the Entropy Cliff.","excerpt":"On ImageNet with $K=16384$, this happens after only about 2 out of 256 positions, when most prefixes already identify individual images in the dataset. We call this phenomenon the Entropy Cliff.","support":"source-stated"},{"text":"The authors propose VCQ, which increases codebook size along the sequence while leaving the loss function, parameter count, and autoregressive training procedure unchanged.","excerpt":"the codebook size $K_t$ grows monotonically along the sequence from $K_{\\min}=2$ to $K_{\\max}$, leaving the loss function, parameter count, and AR training procedure unchanged.","support":"source-stated"},{"text":"On ImageNet 256×256, the authors report gFID w/o CFG of 14.80 for VCQ-Base, compared with 27.98 and 18.16 for constant codebooks of size 16384 and 8192.","excerpt":"VCQ-Base achieves gFID w/o CFG 14.80 on ImageNet $256\\times256$, compared with 27.98 and 18.16 for constant codebooks of size 16384 and 8192, respectively.","support":"source-stated"},{"text":"The authors report that a linear probe using features from the first 10 positions reaches 43.8% top-1 accuracy on ImageNet, versus 27.1% for uniform codebooks.","excerpt":"a linear probe on features from only the first 10 positions reaches 43.8% top-1 accuracy on ImageNet, compared to 27.1% for uniform codebooks.","support":"source-stated"},{"text":"The abstract reports ImageNet results; testing whether the reported results extend to other datasets would establish the scope of the findings.","excerpt":"On ImageNet with $K=16384$, this happens after only about 2 out of 256 positions","support":"interpretation"}],"limitations":["The reported results in the abstract are on ImageNet; it provides no results on other datasets.","The evidence is the paper’s abstract, not its full methods or results."],"mediaId":"3cf45f06dacdca1602ca1d444001d6ddc88e752154a293b3d7dc27d5920460ce","correctionReason":null,"id":"74858ea30a67436d0b488bc23427cedd337835cf22c50ad85841b00686cc1e76","publicationId":"300462da-882e-4136-853a-aea259cdd127","version":1,"sourceUrl":"https://arxiv.org/abs/2605.06207","sourceTitle":"Taming the Entropy Cliff: Variable Codebook Size Quantization for Autoregressive Visual Generation","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-05-07","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:01:00.407Z","createdAt":"2026-11-08T19:23:33.642Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"3cf45f06dacdca1602ca1d444001d6ddc88e752154a293b3d7dc27d5920460ce","kind":"article","url":"https://arxiv.org/abs/2605.06207","sourceUrl":"https://arxiv.org/abs/2605.06207","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-08T19:16:53.823Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2605.06207","checkedAt":"2026-11-08T20:26:08.275Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"3cf45f06dacdca1602ca1d444001d6ddc88e752154a293b3d7dc27d5920460ce","confidence":0.93,"provider":"typesafe","model":"jev-1.13.0","inputHash":"41ee01f5cb29842afa4a60abb26983291006b4305066eaee2449b82cb7c2d63f","decidedAt":"2026-11-08T19:16:54.522Z"},"provenance":{"textHash":"21f23c06b8d9e65224b4fd98ba26a192ae4c489249d5efaacccfa6ffb39ec93a","observationId":"1e98aeb0d799c73534efb56cf4e4f00a432b05bbfad3e155e96b3cf835ffd957","retrievedAt":"2026-11-08T19:16:53.718Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"0b4a3615-aba5-4dd5-8a8d-e4df5d8fe730"},"reviewer":{"model":"gpt-6-luna","runId":"c0fc41ea-5cd6-4208-808b-7a8010256f55"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-08T19:23:33.642Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Zheng, Bowen","url":null,"kind":"unknown"},{"name":"Luo, Weijian","url":null,"kind":"unknown"},{"name":"Yang, Guang","url":null,"kind":"unknown"},{"name":"Zhang, Colin","url":null,"kind":"unknown"},{"name":"Hu, Tianyang","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Zheng, Bowen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zheng, Bowen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.06207","locator":"head/meta[name=author|citation_author]"},"assertionId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:26:08.275Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88","sourceUrl":"https://arxiv.org/abs/2605.06207","contentSha256":"560a6913883d5b2cc3bd148f91cf6686e2457fb25e585ef822fa558a78a3a87f","textHash":"21f23c06b8d9e65224b4fd98ba26a192ae4c489249d5efaacccfa6ffb39ec93a","observedAt":"2026-11-08T20:26:08.275Z"}},{"actor":{"name":"Luo, Weijian","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Luo, Weijian","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.06207","locator":"head/meta[name=author|citation_author]"},"assertionId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:26:08.275Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88","sourceUrl":"https://arxiv.org/abs/2605.06207","contentSha256":"560a6913883d5b2cc3bd148f91cf6686e2457fb25e585ef822fa558a78a3a87f","textHash":"21f23c06b8d9e65224b4fd98ba26a192ae4c489249d5efaacccfa6ffb39ec93a","observedAt":"2026-11-08T20:26:08.275Z"}},{"actor":{"name":"Yang, Guang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yang, Guang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.06207","locator":"head/meta[name=author|citation_author]"},"assertionId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:26:08.275Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88","sourceUrl":"https://arxiv.org/abs/2605.06207","contentSha256":"560a6913883d5b2cc3bd148f91cf6686e2457fb25e585ef822fa558a78a3a87f","textHash":"21f23c06b8d9e65224b4fd98ba26a192ae4c489249d5efaacccfa6ffb39ec93a","observedAt":"2026-11-08T20:26:08.275Z"}},{"actor":{"name":"Zhang, Colin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Colin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.06207","locator":"head/meta[name=author|citation_author]"},"assertionId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:26:08.275Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88","sourceUrl":"https://arxiv.org/abs/2605.06207","contentSha256":"560a6913883d5b2cc3bd148f91cf6686e2457fb25e585ef822fa558a78a3a87f","textHash":"21f23c06b8d9e65224b4fd98ba26a192ae4c489249d5efaacccfa6ffb39ec93a","observedAt":"2026-11-08T20:26:08.275Z"}},{"actor":{"name":"Hu, Tianyang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Hu, Tianyang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.06207","locator":"head/meta[name=author|citation_author]"},"assertionId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:26:08.275Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88","sourceUrl":"https://arxiv.org/abs/2605.06207","contentSha256":"560a6913883d5b2cc3bd148f91cf6686e2457fb25e585ef822fa558a78a3a87f","textHash":"21f23c06b8d9e65224b4fd98ba26a192ae4c489249d5efaacccfa6ffb39ec93a","observedAt":"2026-11-08T20:26:08.275Z"}}],"participationCoverage":{"total":5,"returned":5,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2605.06207","snapshotId":"ff4297a2f2eeaaded816dcf5cccb693cfff9dfcc95b38f36c47130f95ec59c88","contentSha256":"560a6913883d5b2cc3bd148f91cf6686e2457fb25e585ef822fa558a78a3a87f","textHash":"21f23c06b8d9e65224b4fd98ba26a192ae4c489249d5efaacccfa6ffb39ec93a","observedAt":"2026-11-08T20:26:08.275Z","basis":"publisher-page-metadata","sourceLinks":47,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/74858ea30a67436d0b488bc23427cedd337835cf22c50ad85841b00686cc1e76/source"}}}],"audits":[{"publicationId":"300462da-882e-4136-853a-aea259cdd127","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The draft consistently attributes the Entropy Cliff observation, VCQ proposal and benchmark numbers to the paper's authors. It uses 'they report' and 'the authors also report' rather than presenting the claims as established fact.","excerpt":"Authors:Bowen Zheng, Weijian Luo, Guang Yang, Colin Zhang, Tianyang Hu"},{"dimension":"summary","status":"uncertain","explanation":"The summary closely tracks the abstract but leaves central jargon unexplained for a general reader. It never says what a discrete visual tokenizer or codebook is, what gFID measures or that lower is better, what 'w/o CFG' or a linear probe means, or what the authors mean by conditional entropy becoming deterministic. Without this, a reader cannot tell that 14.80 beats 27.98 or why the result matters. Smallest repair: add brief plain-language glosses or term-library tooltips for codebook, tokenizer, gFID (lower is better) and linear probe, keeping the original terms. The title and question are faithful.","excerpt":"Most discrete visual tokenizers rely on a default design: every position in the sequence shares the same codebook."},{"dimension":"date","status":"pass","explanation":"originalPublished 2026-05-07 matches the v1 submission date. The 5 Nov 2026 revision and the November feed date are correctly not used.","excerpt":"[Submitted on 7 May 2026 (v1), last revised 5 Nov 2026 (this version, v2)]"},{"dimension":"support","status":"uncertain","explanation":"The numeric claims match the abstract exactly. However, the limitation that the abstract 'provides no results on other datasets' and the statement that 'the abstract reports these results on ImageNet' omit the abstract's language comparison. In that comparison the authors say entropy declines more gradually, and they use this contrast to support their argument. Repair: say that VCQ's generation results are reported only on ImageNet, and mention the language comparison. The conditional question about generalization to other datasets is otherwise reasonable.","excerpt":"The empirical entropy decreases more gradually in our language comparison, where the effective entropy per position stays below the codebook capacity."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified arXiv abstract page for this paper, which is the direct original source.","excerpt":null}],"createdAt":"2026-11-08T19:23:33.642Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}