# arXiv cs.AI: Vision-language models lag human performance on physical dynamics and intent reasoning

[Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-vision-language-models-lag-human-performance-on-b859be3e)

The paper’s authors argue that vision-language models can perform strongly on controlled benchmarks yet struggle to reason jointly about physical interactions, reference frames and human intentions in open-world environments. They define Teleo-Spatial Intelligence (TSI) as integrating physical dynamics and action goals to reason about actions and their spatial consequences, and introduce EscherVerse to evaluate it. The resource includes real-world videos, an 8,000-example benchmark and an instruction-tuning set. Across 27 models, the strongest proprietary model scored 57.26% overall accuracy; first-pass human performance ranged from 84.81% to 95.14%, with a mean of 90.62%. The authors report that fine-tuning open-weight models on real-world, intent-aware data narrows, but does not close, this gap. Their central point is that recognizing visual patterns is not the same as understanding how physical change relates to what a person is trying to do. EscherVerse is presented as a diagnostic testbed for that purpose-aware spatial reasoning. The abstract reports these comparisons, but does not specify which aspects of the task account for the remaining difference.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv cs.AI · Article

Original source date: Jan 4, 2026 (2026-01-04).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-16T07:03:14.720Z. First observed: 2026-11-16T05:06:37.239Z. Retrieved: 2026-11-16T07:02:16.826Z.

- [Vision-language models lag human performance on physical dynamics and intent reasoning](https://arxiv.org/abs/2601.01547)

## Limits

The evidence provided is the paper’s abstract and publisher page; it does not include the full paper’s methods or detailed comparison conditions. The abstract reports that fine-tuning narrows the gap but does not specify which aspects of the tasks account for the remaining difference.


## Open question

Which aspects of physical dynamics, reference frames or inferred human intentions account for the remaining gap after fine-tuning?


## Source-stated

The authors define Teleo-Spatial Intelligence as integrating physical dynamics and action goals to reason about actions and their spatial consequences.

> integrating physical dynamics and action goals to reason about actions and their spatial consequences

- [Supporting source](https://arxiv.org/abs/2601.01547)

## Source-stated

EscherVerse includes an 8,000-example benchmark and a 35,963-example instruction-tuning set, built from 11,328 real-world videos.

> built from 11,328 real-world videos, including an 8,000-example benchmark and a 35,963-example instruction-tuning set

- [Supporting source](https://arxiv.org/abs/2601.01547)

## Source-stated

Across 27 vision-language models, the strongest proprietary model achieved 57.26% overall accuracy; first-pass human performance ranged from 84.81% to 95.14%, with a mean of 90.62%.

> the strongest proprietary model achieves 57.26% overall accuracy, far below first-pass human performance, which ranges from 84.81% to 95.14% with a mean of 90.62%

- [Supporting source](https://arxiv.org/abs/2601.01547)

## Source-stated

The authors report that fine-tuning open-weight models on real-world, intent-aware data narrows but does not close the reported gap.

> Fine-tuning on real-world, intent-aware data narrows this gap for open-weight models, but does not close it.

- [Supporting source](https://arxiv.org/abs/2601.01547)

## Source and contributor context

Gu, Tianjun, Gong, Jingyu, Zhang, Zhizhong, Xie, Yuan, Ma, Lizhuang, Tan, Xin, Vasilakos, Athanasios V. · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

- [Definition reference](https://arxiv.org/abs/2211.09110)

## Fine-tuning

Further training of an existing model to adapt it to a particular task or use.

- [Definition reference](https://developers.google.com/machine-learning/glossary#fine-tuning)

## Open weights

A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.

- [Definition reference](https://opensource.org/ai/open-weights)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: c2681b91-4d0e-4731-8034-995ef9617d72.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-16T07:03:14.720Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: fbe5364046e7386b4577297bec7ca7b06fad65dd85b1db4d7662428174944a98. Source text hash: d8cf26837c5d2ccb9269e3313f9763c123c9f363b53980fd80544b174f25d935.


## Revision and source-check history

### Version 1 — 2026-11-16T07:03:14.720Z

The paper’s authors argue that vision-language models can perform strongly on controlled benchmarks yet struggle to reason jointly about physical interactions, reference frames and human intentions in open-world environments. They define Teleo-Spatial Intelligence (TSI) as integrating physical dynamics and action goals to reason about actions and their spatial consequences, and introduce EscherVerse to evaluate it. The resource includes real-world videos, an 8,000-example benchmark and an instruction-tuning set. Across 27 models, the strongest proprietary model scored 57.26% overall accuracy; first-pass human performance ranged from 84.81% to 95.14%, with a mean of 90.62%. The authors report that fine-tuning open-weight models on real-world, intent-aware data narrows, but does not close, this gap. Their central point is that recognizing visual patterns is not the same as understanding how physical change relates to what a person is trying to do. EscherVerse is presented as a diagnostic testbed for that purpose-aware spatial reasoning. The abstract reports these comparisons, but does not specify which aspects of the task account for the remaining difference.

## Original source

arXiv cs.AI · Article

Original source date: Jan 4, 2026 (2026-01-04).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-16T07:03:14.720Z. First observed: 2026-11-16T05:06:37.239Z. Retrieved: 2026-11-16T07:02:16.826Z.

- [Vision-language models lag human performance on physical dynamics and intent reasoning](https://arxiv.org/abs/2601.01547)

## Limits

The evidence provided is the paper’s abstract and publisher page; it does not include the full paper’s methods or detailed comparison conditions. The abstract reports that fine-tuning narrows the gap but does not specify which aspects of the tasks account for the remaining difference.


## Open question

Which aspects of physical dynamics, reference frames or inferred human intentions account for the remaining gap after fine-tuning?


## Source-stated

The authors define Teleo-Spatial Intelligence as integrating physical dynamics and action goals to reason about actions and their spatial consequences.

> integrating physical dynamics and action goals to reason about actions and their spatial consequences

- [Supporting source](https://arxiv.org/abs/2601.01547)

## Source-stated

EscherVerse includes an 8,000-example benchmark and a 35,963-example instruction-tuning set, built from 11,328 real-world videos.

> built from 11,328 real-world videos, including an 8,000-example benchmark and a 35,963-example instruction-tuning set

- [Supporting source](https://arxiv.org/abs/2601.01547)

## Source-stated

Across 27 vision-language models, the strongest proprietary model achieved 57.26% overall accuracy; first-pass human performance ranged from 84.81% to 95.14%, with a mean of 90.62%.

> the strongest proprietary model achieves 57.26% overall accuracy, far below first-pass human performance, which ranges from 84.81% to 95.14% with a mean of 90.62%

- [Supporting source](https://arxiv.org/abs/2601.01547)

## Source-stated

The authors report that fine-tuning open-weight models on real-world, intent-aware data narrows but does not close the reported gap.

> Fine-tuning on real-world, intent-aware data narrows this gap for open-weight models, but does not close it.

- [Supporting source](https://arxiv.org/abs/2601.01547)

## Source and contributor context

Gu, Tianjun, Gong, Jingyu, Zhang, Zhizhong, Xie, Yuan, Ma, Lizhuang, Tan, Xin, Vasilakos, Athanasios V. · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

- [Definition reference](https://arxiv.org/abs/2211.09110)

## Fine-tuning

Further training of an existing model to adapt it to a particular task or use.

- [Definition reference](https://developers.google.com/machine-learning/glossary#fine-tuning)

## Open weights

A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.

- [Definition reference](https://opensource.org/ai/open-weights)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: c2681b91-4d0e-4731-8034-995ef9617d72.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-16T07:03:14.720Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: fbe5364046e7386b4577297bec7ca7b06fad65dd85b1db4d7662428174944a98. Source text hash: d8cf26837c5d2ccb9269e3313f9763c123c9f363b53980fd80544b174f25d935.


### passed — 2026-11-16T07:03:14.720Z — anthropic/claude-opus-5.5

Publication: c2681b91-4d0e-4731-8034-995ef9617d72

attribution — pass: The reading attributes the argument, the TSI definition, the EscherVerse resource and the results to the paper's authors. arXiv is presented only as the publisher. Reported findings stay framed as the authors' claims, not as Escape Meridian conclusions.

> Authors:Tianjun Gu, Jingyu Gong, Zhizhong Zhang, Yuan Xie, Lizhuang Ma, Xin Tan, Athanasios V. Vasilakos

summary — pass: The summary leads with the authors' argument and explains TSI in plain terms. It gives the key benchmark comparison and the fine-tuning result, and it preserves the 'narrows but does not close' condition. The pattern-recognition versus understanding point closely tracks the abstract. The question names a real unresolved issue the abstract leaves open. Terms such as vision-language model and open-weight are left as technical terms for tooltips. Optional additions would be that the human baseline came from 11 annotators and that the abstract frames this in terms of embodied AI.

> highlights a critical gap between pattern recognition and human-level understanding in embodied AI

date — pass: The displayed date of 2026-01-04 matches the v1 submission date. It is not the date of the later v3 revision (13 Nov 2026) or the feed entry date.

> \[Submitted on 4 Jan 2026 (v1), last revised 13 Nov 2026 (this version, v3)\]

support — pass: Every claim's excerpt matches the source exactly, and the numbers are reproduced correctly. The reading does not turn the benchmark scores into claims about real-world reliability. The limitations accurately note that only the abstract and page are available. The question follows from the source and implies no unsupported benefits or costs.

> Fine-tuning on real-world, intent-aware data narrows this gap for open-weight models, but does not close it.

media — pass: The selected media is the verified direct link to the original arXiv abstract page. It matches the source URL and is the most direct way for readers to reach the paper.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/b859be3e92ff278b614a9b16be5c36b937a251b22f8856950c4e0619812a882f/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-vision-language-models-lag-human-performance-on-b859be3e.json)
