arXiv cs.AI: Playing Dumb: How a Chess Transformer Simulates Users Across Skill Levels

arXiv cs.AI: Playing Dumb: How a Chess Transformer Simulates Users Across Skill Levels

Difan Jiao, George Eilender, Zhenwei Tang and Ashton Anderson study Maia-2, a chess transformer that predicts human moves conditioned on player skill. They ask whether weaker play reflects less knowledge or a different use of knowledge the model retains. The authors report that probes for 172 chess concepts found nearly constant accuracy across skill conditions, and that probes trained at one level transferred to others. In positions where skill conditioning changed a prediction from a blunder to the best move, fine-tuning only the final output layer raised best-move accuracy in the lowest skill band from 14.4% to 82.8%. The authors also report that training on highest-skill representations retained most of the improvement at lower levels. They interpret these results as evidence that skill conditioning changes how retained concept knowledge influences move selection, rather than simply removing that knowledge. This distinction matters for interpreting simulated low-skill behavior: in this chess model, weaker choices may understate knowledge available internally. Whether similar results hold beyond chess, or for other ways of measuring knowledge, remains an open question; the evidence here concerns Maia-2 and chess concepts.

Publication: 4e9dd0d8-b4c8-4fe4-8b15-376674e37811

attribution — pass: The reading correctly names authors Difan Jiao, George Eilender, Zhenwei Tang and Ashton Anderson. It attributes the findings and interpretation to them with phrases such as 'the authors report' and 'they interpret', rather than presenting them as established fact.

Authors:Difan Jiao, George Eilender, Zhenwei Tang, Ashton Anderson

summary — uncertain: The reading is faithful but may be hard for a non-specialist to follow. 'Probes' carries the central knowledge-retention evidence yet is never explained; 'fine-tuning', 'skill conditioning' and 'transformer' are also unexplained. It also omits the stakes the abstract gives: generative models are used to simulate users of different skill levels in education and AI evaluation. Smallest repair: briefly gloss 'probes' (small classifiers that read concepts from internal states) or add tooltips, and add one sentence on the education/evaluation use. The sparse autoencoder steering result is also omitted, though that is minor.

Generative models increasingly simulate users at different skill levels in education and AI evaluation.

date — pass: originalPublished is 2026-11-02, which matches the arXiv v1 submission date. The feed date of Nov 3 is later and correctly not used.

[Submitted on 2 Nov 2026]

support — pass: Every claim matches an abstract excerpt. This includes the 172 concepts, the near-constant probe accuracy and transfer, the rise from 14.4% to 82.8%, the retained gains from training on highest-skill representations, and the authors' interpretation. The generalization question and the limitations are framed as unknowns, not as claims. No implied benefits or costs go beyond the evidence.

Probe accuracy remains nearly constant across skill conditions, and probes trained at one level transfer accurately to the others.

media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.