Liu, Yin · Nov 6, 2026
arXiv cs.AI: Seeing vs. Counting: Complexity Twins Show Where Multimodal LLMs Fail on Diagrams
Markdown · JSON · Original source
Yin Liu argues that diagram-counting errors in can come from two different failures: misreading the drawing or computing the answer. Existing often change visual density and algorithmic hardness together, making those causes difficult to separate. Liu pairs counting questions about the same drawing that belong to different counting-complexity classes, and uses a transcription probe and a text-only control to distinguish reading from computing. In the reported tests, most Gemini models in read small drawings correctly but then failed to count. For the primary model, exact-answer odds fell 2.38-fold per decade of answer size; the paper says answer size mattered far more than complexity class. With code and an exact transcription, that model counted correctly even on 10x10 grids, shifting the bottleneck to reading the diagram. A second reading of a rotated copy with crop checks read 86/96 large grids exactly, compared with 57/96 when the upright reading was repeated. The results suggest that a model’s apparent counting weakness can depend on whether it can read the diagram and how it computes, not just on the counting problem’s complexity.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 6, 2026 (2026-11-06).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T05:33:40.627Z. First observed: 2026-11-09T05:03:47.062Z. Retrieved: 2026-11-09T05:32:46.602Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The supplied evidence is the abstract and page text, not the paper’s full methods or results; it does not provide details needed to assess the experiments beyond the abstract. The reported findings concern the models and diagram tasks tested; the supplied evidence does not establish whether they generalize to other models or diagrams.
Open question
Would the separation between reading and counting hold for models and diagrams beyond those tested?
Source-stated
Liu says existing benchmarks can confound visual reading with computational difficulty because they vary visual density and algorithmic hardness together.
Existing visual graph and puzzle benchmarks vary visual density and algorithmic hardness together, confounding the two.
Source-stated
The study pairs counting questions about the same drawing from different counting-complexity classes, using a transcription probe and a text-only control to distinguish reading from computing.
We pair counting questions about the same drawing that sit in different counting-complexity classes
Source-stated
For the primary model, the reported odds of an exact answer fell 2.38-fold per decade of answer size, and answer size mattered far more than complexity class.
the odds of an exact answer fall 2.38-fold per decade of the answer
Source-stated
With code and an exact transcription, the primary model counted correctly on 10x10 grids; the abstract then identifies reading as the bottleneck.
Given code and an exact transcription, the primary model counts correctly even on 10x10 grids, and reading becomes the bottleneck
Source-stated
A rotated second reading with crop checks read 86 of 96 large grids exactly, compared with 57 of 96 when the upright reading was repeated.
a second reading of a rotated copy with crop checks reads 86/96 large grids exactly, against 57/96 when the upright reading is simply repeated.
Source and contributor context
Liu, Yin · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Multimodal model
A model that works with more than one kind of input or output, such as text, images and audio.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Chain of thought
The step-by-step working a model writes out before giving its final answer.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: e7c6fd35-831f-45c8-8f62-3442643c601d.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-09T05:33:40.627Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 545ccfde0b8c7f3092913c521920cbc19a7e9ef3957a5b4edcc13a7b217d9d4f. Source text hash: 11aaaf2215f391b8b66e163444f81753aa47493e1e835f89909764810b556372.
Revision and source-check history
Version 1 · 2026-11-09T05:33:40.627Z
arXiv cs.AI: Seeing vs. Counting: Complexity Twins Show Where Multimodal LLMs Fail on Diagrams
Yin Liu argues that diagram-counting errors in can come from two different failures: misreading the drawing or computing the answer. Existing often change visual density and algorithmic hardness together, making those causes difficult to separate. Liu pairs counting questions about the same drawing that belong to different counting-complexity classes, and uses a transcription probe and a text-only control to distinguish reading from computing. In the reported tests, most Gemini models in read small drawings correctly but then failed to count. For the primary model, exact-answer odds fell 2.38-fold per decade of answer size; the paper says answer size mattered far more than complexity class. With code and an exact transcription, that model counted correctly even on 10x10 grids, shifting the bottleneck to reading the diagram. A second reading of a rotated copy with crop checks read 86/96 large grids exactly, compared with 57/96 when the upright reading was repeated. The results suggest that a model’s apparent counting weakness can depend on whether it can read the diagram and how it computes, not just on the counting problem’s complexity.
Original source
cs.AI · Article
Source details
Original source date: Nov 6, 2026 (2026-11-06).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T05:33:40.627Z. First observed: 2026-11-09T05:03:47.062Z. Retrieved: 2026-11-09T05:32:46.602Z.
Limits
The supplied evidence is the abstract and page text, not the paper’s full methods or results; it does not provide details needed to assess the experiments beyond the abstract. The reported findings concern the models and diagram tasks tested; the supplied evidence does not establish whether they generalize to other models or diagrams.
Open question
Would the separation between reading and counting hold for models and diagrams beyond those tested?
Source-stated
Liu says existing benchmarks can confound visual reading with computational difficulty because they vary visual density and algorithmic hardness together.
Existing visual graph and puzzle benchmarks vary visual density and algorithmic hardness together, confounding the two.
Source-stated
The study pairs counting questions about the same drawing from different counting-complexity classes, using a transcription probe and a text-only control to distinguish reading from computing.
We pair counting questions about the same drawing that sit in different counting-complexity classes
Source-stated
For the primary model, the reported odds of an exact answer fell 2.38-fold per decade of answer size, and answer size mattered far more than complexity class.
the odds of an exact answer fall 2.38-fold per decade of the answer
Source-stated
With code and an exact transcription, the primary model counted correctly on 10x10 grids; the abstract then identifies reading as the bottleneck.
Given code and an exact transcription, the primary model counts correctly even on 10x10 grids, and reading becomes the bottleneck
Source-stated
A rotated second reading with crop checks read 86 of 96 large grids exactly, compared with 57 of 96 when the upright reading was repeated.
a second reading of a rotated copy with crop checks reads 86/96 large grids exactly, against 57/96 when the upright reading is simply repeated.
Source and contributor context
Liu, Yin · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Multimodal model
A model that works with more than one kind of input or output, such as text, images and audio.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Chain of thought
The step-by-step working a model writes out before giving its final answer.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: e7c6fd35-831f-45c8-8f62-3442643c601d.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-09T05:33:40.627Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 545ccfde0b8c7f3092913c521920cbc19a7e9ef3957a5b4edcc13a7b217d9d4f. Source text hash: 11aaaf2215f391b8b66e163444f81753aa47493e1e835f89909764810b556372.
passed · 2026-11-09T05:33:40.627Z · anthropic/claude-opus-5.5
Publication: e7c6fd35-831f-45c8-8f62-3442643c601d
attribution — pass: The reading attributes the study and its claims to Yin Liu, the sole listed author of the arXiv paper. Results are framed as reported or as what the paper says, not as established fact. The 'arXiv cs.AI' title prefix matches the supplied publisher field, although the paper's primary listing is cs.CV with cs.AI as a cross-list.
Authors:Yin Liu
summary — pass: The summary leads with Liu's central question: is a miscount a failure of seeing or of counting? It explains why existing benchmarks confound the two, describes the paired-question design and reports the key results faithfully. Terms such as chain of thought, counting-complexity classes and transcription probe are kept exactly, which suits tooltips. Optional repairs: note that Gemma and Qwen3-VL were also tested, name the spanning-tree versus simple-path example, and add that a single view misses horizontal edges, which explains why the rotated reading helps.
When a multimodal LLM miscounts in a diagram, does it fail at seeing or at counting?
date — pass: The displayed original publication date, 2026-11-06, matches the v1 submission date on arXiv. The later feed date of 2026-11-09 is correctly not used.
[Submitted on 6 Nov 2026]
support — pass: Every claim is backed by verbatim excerpts: the confound, the design, the 2.38-fold odds figure, correct counting with code and an exact transcription, and the 86/96 versus 57/96 comparison. The closing interpretive sentence stays close to the abstract. The limitations accurately note that only the abstract was supplied and that the findings cover only the tested models and tasks. The question about generalization is open and not leading.
for the Gemini models, the answer depends on how the model computes
media — pass: The selected media is the verified direct link to the original arXiv abstract page, the most direct access to the source. No other candidate media was available.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A model that works with more than one kind of input or output, such as text, images and audio.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceThe step-by-step working a model writes out before giving its final answer.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA model that works with more than one kind of input or output, such as text, images and audio.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceThe step-by-step working a model writes out before giving its final answer.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv