Melanie Mitchell · Oct 6, 2025
Melanie Mitchell argues that accuracy can misread abstract reasoning
Markdown · JSON · Original source
Melanie Mitchell’s source is dated October 6, 2025. She describes her group’s study of on ConceptARC tasks, designed to isolate core concepts. The group compared models’ answers and stated rules. Mitchell reports that with textual inputs, models approached or exceeded human accuracy but used unintended shortcuts more often than humans. With visual inputs, they generated accurate grids poorly, yet produced intended rules more often than correct grids. A rule can fit examples without capturing the abstraction a task is meant to test. Mitchell argues that accuracy alone can misrepresent abstract reasoning, depending on the input type. She says humanlike abstractions help people handle novelty and predict outcomes, and that AI systems need to grasp human intentions and explain their reasoning understandably. Her conclusion is that evaluations need to go beyond accuracy. The account does not establish how well these findings extend beyond the tested tasks and inputs.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
Melanie Mitchell · Article
Source details
Original source date: Oct 6, 2025 (2025-10-06T21:00:25.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-11T20:43:23.872Z. First observed: 2026-10-23T23:46:08.144Z. Retrieved: 2026-10-23T23:47:38.287Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
Text as retrieved on October 23, 2026; originally published October 6, 2025. The account describes results on ConceptARC tasks and the tested input types; it does not establish how the findings extend to other tasks or real-world settings.
Open question
How well do these findings about ConceptARC tasks extend to other evaluations of abstract reasoning?
Source-stated
Melanie Mitchell says her group evaluated reasoning models on ConceptARC tasks designed to isolate core concepts.
we investigated this question by evaluating several reasoning models, including OpenAI’s o3, Anthropic’s Claude Sonnet 4, and Google’s Gemini 2.5 Pro, on 480 tasks from ConceptARC
Source-stated
Mitchell reports that models approached or exceeded human accuracy with textual inputs but were more prone than humans to use unintended shortcuts.
We found that while o3, Claude, and Gemini approach or exceed human accuracy on these tasks with textual inputs, they are substantially more prone to use unintended “shortcuts” to solve the tasks than do humans.
Source-stated
Mitchell reports that with visual inputs the models generated accurate grids poorly, while more often producing intended rules than correct grids.
With visual inputs, these models all do quite poorly on generating accurate grids. However, they do generate “correct as intended” rules more often than they generate correct grids
Source-stated
Mitchell argues that accuracy alone can overestimate abstract reasoning in the textual setting and underestimate it in the visual setting.
evaluations like those done for the ARC Prize, using accuracy alone, may be overestimating abstract reasoning ability of these models in the textual setting. On the other hand, using accuracy alone may be underestimating the ability for abstract reasoning in the visual setting.
Source-stated
Mitchell says humanlike abstractions matter for handling novelty and for AI systems to understand human intentions and explain reasoning understandably.
These are all things we want AI systems to do in more trustworthy ways. Moreover, if we humans are to work together with AI systems, it’s essential to develop AI models that are better at grasping our abstractions
Source and contributor context
Melanie Mitchell · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Reasoning model
A language model trained to work through a problem in written steps before it answers.
Editorial provenance
Version 1. Publication ID: 848f9634-2f85-ba2c-0f0f-e0bdd5a474e6.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T20:40:01.988Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model, second_opinion_categorical.
Observation: ce7a145cacd345185f7ae8b69fcf22a62e248a74c7a0dd7508f41dd2c426358f. Source text hash: 55b8086d47eced7cc97ffa494ba6c7622e44b5abea60cb274973b566ef66b0a1.
Revision and source-check history
Version 1 · 2026-11-11T20:43:23.872Z
Melanie Mitchell argues that accuracy can misread abstract reasoning
Melanie Mitchell’s source is dated October 6, 2025. She describes her group’s study of on ConceptARC tasks, designed to isolate core concepts. The group compared models’ answers and stated rules. Mitchell reports that with textual inputs, models approached or exceeded human accuracy but used unintended shortcuts more often than humans. With visual inputs, they generated accurate grids poorly, yet produced intended rules more often than correct grids. A rule can fit examples without capturing the abstraction a task is meant to test. Mitchell argues that accuracy alone can misrepresent abstract reasoning, depending on the input type. She says humanlike abstractions help people handle novelty and predict outcomes, and that AI systems need to grasp human intentions and explain their reasoning understandably. Her conclusion is that evaluations need to go beyond accuracy. The account does not establish how well these findings extend beyond the tested tasks and inputs.
Original source
Melanie Mitchell · Article
Source details
Original source date: Oct 6, 2025 (2025-10-06T21:00:25.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-11T20:43:23.872Z. First observed: 2026-10-23T23:46:08.144Z. Retrieved: 2026-10-23T23:47:38.287Z.
Limits
Text as retrieved on October 23, 2026; originally published October 6, 2025. The account describes results on ConceptARC tasks and the tested input types; it does not establish how the findings extend to other tasks or real-world settings.
Open question
How well do these findings about ConceptARC tasks extend to other evaluations of abstract reasoning?
Source-stated
Melanie Mitchell says her group evaluated reasoning models on ConceptARC tasks designed to isolate core concepts.
we investigated this question by evaluating several reasoning models, including OpenAI’s o3, Anthropic’s Claude Sonnet 4, and Google’s Gemini 2.5 Pro, on 480 tasks from ConceptARC
Source-stated
Mitchell reports that models approached or exceeded human accuracy with textual inputs but were more prone than humans to use unintended shortcuts.
We found that while o3, Claude, and Gemini approach or exceed human accuracy on these tasks with textual inputs, they are substantially more prone to use unintended “shortcuts” to solve the tasks than do humans.
Source-stated
Mitchell reports that with visual inputs the models generated accurate grids poorly, while more often producing intended rules than correct grids.
With visual inputs, these models all do quite poorly on generating accurate grids. However, they do generate “correct as intended” rules more often than they generate correct grids
Source-stated
Mitchell argues that accuracy alone can overestimate abstract reasoning in the textual setting and underestimate it in the visual setting.
evaluations like those done for the ARC Prize, using accuracy alone, may be overestimating abstract reasoning ability of these models in the textual setting. On the other hand, using accuracy alone may be underestimating the ability for abstract reasoning in the visual setting.
Source-stated
Mitchell says humanlike abstractions matter for handling novelty and for AI systems to understand human intentions and explain reasoning understandably.
These are all things we want AI systems to do in more trustworthy ways. Moreover, if we humans are to work together with AI systems, it’s essential to develop AI models that are better at grasping our abstractions
Source and contributor context
Melanie Mitchell · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Reasoning model
A language model trained to work through a problem in written steps before it answers.
Editorial provenance
Version 1. Publication ID: 848f9634-2f85-ba2c-0f0f-e0bdd5a474e6.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T20:40:01.988Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model, second_opinion_categorical.
Observation: ce7a145cacd345185f7ae8b69fcf22a62e248a74c7a0dd7508f41dd2c426358f. Source text hash: 55b8086d47eced7cc97ffa494ba6c7622e44b5abea60cb274973b566ef66b0a1.
uncertain · 2026-11-11T20:40:01.988Z · anthropic/claude-opus-5.5
Publication: 848f9634-2f85-ba2c-0f0f-e0bdd5a474e6
attribution — pass: The reading attributes the post and the study to Melanie Mitchell and her group. This matches the byline and her statement that she is summarizing a new paper from her group. It is one authored piece, not a listing.
Here I’ll summarize a new paper from my group
summary — uncertain: The summary is faithful, but a non-specialist cannot tell what the tasks are. It refers to models generating "accurate grids" and "correct grids" without explaining that ConceptARC tasks show example input grids and ask for the output grid. It also does not say that textual inputs are grids written as number matrices while visual inputs are images. Smallest repair: one sentence saying tasks show example colored-grid transformations and ask for the output grid, given as numbers (text) or images (visual). The phrase "depending on the input type" also blurs the direction of the effect: overestimation for text, underestimation for visuals.
The task gives three demonstrations of a transformation rule that maps an input grid to an output grid.
date — pass: October 6, 2025 matches the dateline and the UTC day of originalPublished. The modified date equals the publication date. Nothing is presented as known after that date, and the page is a publisher page, so no archive disclosure is needed.
Melanie Mitchell Oct 06, 2025
support — pass: Each claim matches its quoted source text: textual-input accuracy combined with more shortcuts, poor visual grids alongside more intended rules than correct grids, and the over- and under-estimation argument. The reasons why abstractions matter are also supported. The limitation and the question are framed as open, not asserted.
they are substantially more prone to use unintended “shortcuts” to solve the tasks than do humans
media — pass: No media was selected, and no verified candidates were supplied, so nothing is misidentified.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A language model trained to work through a problem in written steps before it answers.
Definition referenceA language model trained to work through a problem in written steps before it answers.
Definition reference