{"id":"cad9ba4cfc10c0fbd3803e5be856a06ed88c23efb48526adaf419495a84797a8","title":"Melanie Mitchell argues that accuracy can misread abstract reasoning","argument":"Melanie Mitchell’s source is dated October 6, 2025. She describes her group’s study of reasoning models on ConceptARC tasks, designed to isolate core concepts. The group compared models’ answers and stated rules. Mitchell reports that with textual inputs, models approached or exceeded human accuracy but used unintended shortcuts more often than humans. With visual inputs, they generated accurate grids poorly, yet produced intended rules more often than correct grids. A rule can fit examples without capturing the abstraction a task is meant to test. Mitchell argues that accuracy alone can misrepresent abstract reasoning, depending on the input type. She says humanlike abstractions help people handle novelty and predict outcomes, and that AI systems need to grasp human intentions and explain their reasoning understandably. Her conclusion is that evaluations need to go beyond accuracy. The account does not establish how well these findings extend beyond the tested tasks and inputs.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 23, 2026; originally published October 6, 2025. The account describes results on ConceptARC tasks and the tested input types; it does not establish how the findings extend to other tasks or real-world settings.","speaker":{"name":"Melanie Mitchell","roleAtSource":"Publisher"},"source":{"title":"Do AI Reasoning Models Abstract and Reason Like Humans?","publisher":"Melanie Mitchell","format":"Article","url":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and","publishedAt":"2025-10-06T21:00:25.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"Melanie Mitchell says her group evaluated reasoning models on ConceptARC tasks designed to isolate core concepts.","excerpt":"we investigated this question by evaluating several reasoning models, including OpenAI’s o3, Anthropic’s Claude Sonnet 4, and Google’s Gemini 2.5 Pro, on 480 tasks from ConceptARC","url":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and","relationship":"Source-stated"},{"text":"Mitchell reports that models approached or exceeded human accuracy with textual inputs but were more prone than humans to use unintended shortcuts.","excerpt":"We found that while o3, Claude, and Gemini approach or exceed human accuracy on these tasks with textual inputs, they are substantially more prone to use unintended “shortcuts” to solve the tasks than do humans.","url":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and","relationship":"Source-stated"},{"text":"Mitchell reports that with visual inputs the models generated accurate grids poorly, while more often producing intended rules than correct grids.","excerpt":"With visual inputs, these models all do quite poorly on generating accurate grids. However, they do generate “correct as intended” rules more often than they generate correct grids","url":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and","relationship":"Source-stated"},{"text":"Mitchell argues that accuracy alone can overestimate abstract reasoning in the textual setting and underestimate it in the visual setting.","excerpt":"evaluations like those done for the ARC Prize, using accuracy alone, may be overestimating abstract reasoning ability of these models in the textual setting. On the other hand, using accuracy alone may be underestimating the ability for abstract reasoning in the visual setting.","url":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and","relationship":"Source-stated"},{"text":"Mitchell says humanlike abstractions matter for handling novelty and for AI systems to understand human intentions and explain reasoning understandably.","excerpt":"These are all things we want AI systems to do in more trustworthy ways.\nMoreover, if we humans are to work together with AI systems, it’s essential to develop AI models that are better at grasping our abstractions","url":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well do these findings about ConceptARC tasks extend to other evaluations of abstract reasoning?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Melanie Mitchell argues that accuracy can misread abstract reasoning","summary":"Melanie Mitchell’s source is dated October 6, 2025. She describes her group’s study of reasoning models on ConceptARC tasks, designed to isolate core concepts. The group compared models’ answers and stated rules. Mitchell reports that with textual inputs, models approached or exceeded human accuracy but used unintended shortcuts more often than humans. With visual inputs, they generated accurate grids poorly, yet produced intended rules more often than correct grids. A rule can fit examples without capturing the abstraction a task is meant to test. Mitchell argues that accuracy alone can misrepresent abstract reasoning, depending on the input type. She says humanlike abstractions help people handle novelty and predict outcomes, and that AI systems need to grasp human intentions and explain their reasoning understandably. Her conclusion is that evaluations need to go beyond accuracy. The account does not establish how well these findings extend beyond the tested tasks and inputs.","question":"How well do these findings about ConceptARC tasks extend to other evaluations of abstract reasoning?","claims":[{"text":"Melanie Mitchell says her group evaluated reasoning models on ConceptARC tasks designed to isolate core concepts.","excerpt":"we investigated this question by evaluating several reasoning models, including OpenAI’s o3, Anthropic’s Claude Sonnet 4, and Google’s Gemini 2.5 Pro, on 480 tasks from ConceptARC","support":"source-stated"},{"text":"Mitchell reports that models approached or exceeded human accuracy with textual inputs but were more prone than humans to use unintended shortcuts.","excerpt":"We found that while o3, Claude, and Gemini approach or exceed human accuracy on these tasks with textual inputs, they are substantially more prone to use unintended “shortcuts” to solve the tasks than do humans.","support":"source-stated"},{"text":"Mitchell reports that with visual inputs the models generated accurate grids poorly, while more often producing intended rules than correct grids.","excerpt":"With visual inputs, these models all do quite poorly on generating accurate grids. However, they do generate “correct as intended” rules more often than they generate correct grids","support":"source-stated"},{"text":"Mitchell argues that accuracy alone can overestimate abstract reasoning in the textual setting and underestimate it in the visual setting.","excerpt":"evaluations like those done for the ARC Prize, using accuracy alone, may be overestimating abstract reasoning ability of these models in the textual setting. On the other hand, using accuracy alone may be underestimating the ability for abstract reasoning in the visual setting.","support":"source-stated"},{"text":"Mitchell says humanlike abstractions matter for handling novelty and for AI systems to understand human intentions and explain reasoning understandably.","excerpt":"These are all things we want AI systems to do in more trustworthy ways.\nMoreover, if we humans are to work together with AI systems, it’s essential to develop AI models that are better at grasping our abstractions","support":"source-stated"}],"limitations":["Text as retrieved on October 23, 2026; originally published October 6, 2025.","The account describes results on ConceptARC tasks and the tested input types; it does not establish how the findings extend to other tasks or real-world settings."],"mediaId":null,"correctionReason":null,"id":"cad9ba4cfc10c0fbd3803e5be856a06ed88c23efb48526adaf419495a84797a8","publicationId":"848f9634-2f85-ba2c-0f0f-e0bdd5a474e6","version":1,"sourceUrl":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and","sourceTitle":"Do AI Reasoning Models Abstract and Reason Like Humans?","publisher":"Melanie Mitchell","publishedAt":"2025-10-06T21:00:25.000Z","originalPublished":{"value":"2025-10-06T21:00:25.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-23T23:46:08.144Z","createdAt":"2026-11-11T20:43:23.872Z","historical":true,"entityIds":[],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"88121d7f9cf66b3eb9164118f610cef1c01b8d277a0ed63a3a84bf336017d468","rubricVersion":"unrecorded","provider":"unrecorded","model":"anthropic/claude-opus-5.5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-23T23:46:04.042Z"},"provenance":{"textHash":"55b8086d47eced7cc97ffa494ba6c7622e44b5abea60cb274973b566ef66b0a1","observationId":"ce7a145cacd345185f7ae8b69fcf22a62e248a74c7a0dd7508f41dd2c426358f","retrievedAt":"2026-10-23T23:47:38.287Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"a036b2d1-4db5-77cd-814b-4735fa3b5731"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"f93eb0fc-acf2-c931-aa43-bc6f48a45dee"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-11T20:40:01.988Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/melanie-mitchell-argues-that-accuracy-can-misread-abstract-cad9ba4c","formats":{"html":"/readings/melanie-mitchell-argues-that-accuracy-can-misread-abstract-cad9ba4c","markdown":"/readings/melanie-mitchell-argues-that-accuracy-can-misread-abstract-cad9ba4c.md","json":"/readings/melanie-mitchell-argues-that-accuracy-can-misread-abstract-cad9ba4c.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-reasoning-model","name":"Reasoning model","explanation":"A language model trained to work through a problem in written steps before it answers.","referenceUrl":"https://en.wikipedia.org/wiki/Reasoning_model"}],"entities":[]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"cad9ba4cfc10c0fbd3803e5be856a06ed88c23efb48526adaf419495a84797a8","revision":"848f9634-2f85-ba2c-0f0f-e0bdd5a474e6","revisionKind":"publication-id","title":"Melanie Mitchell argues that accuracy can misread abstract reasoning","url":"https://escapemeridian.com/readings/melanie-mitchell-argues-that-accuracy-can-misread-abstract-cad9ba4c"},"source":{"type":"source","id":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and","revision":"55b8086d47eced7cc97ffa494ba6c7622e44b5abea60cb274973b566ef66b0a1","revisionKind":"sha256-source-text","title":"Do AI Reasoning Models Abstract and Reason Like Humans?","url":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"cad9ba4cfc10c0fbd3803e5be856a06ed88c23efb48526adaf419495a84797a8","title":"Melanie Mitchell argues that accuracy can misread abstract reasoning","argument":"Melanie Mitchell’s source is dated October 6, 2025. She describes her group’s study of reasoning models on ConceptARC tasks, designed to isolate core concepts. The group compared models’ answers and stated rules. Mitchell reports that with textual inputs, models approached or exceeded human accuracy but used unintended shortcuts more often than humans. With visual inputs, they generated accurate grids poorly, yet produced intended rules more often than correct grids. A rule can fit examples without capturing the abstraction a task is meant to test. Mitchell argues that accuracy alone can misrepresent abstract reasoning, depending on the input type. She says humanlike abstractions help people handle novelty and predict outcomes, and that AI systems need to grasp human intentions and explain their reasoning understandably. Her conclusion is that evaluations need to go beyond accuracy. The account does not establish how well these findings extend beyond the tested tasks and inputs.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 23, 2026; originally published October 6, 2025. The account describes results on ConceptARC tasks and the tested input types; it does not establish how the findings extend to other tasks or real-world settings.","speaker":{"name":"Melanie Mitchell","roleAtSource":"Publisher"},"source":{"title":"Do AI Reasoning Models Abstract and Reason Like Humans?","publisher":"Melanie Mitchell","format":"Article","url":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and","publishedAt":"2025-10-06T21:00:25.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"Melanie Mitchell says her group evaluated reasoning models on ConceptARC tasks designed to isolate core concepts.","excerpt":"we investigated this question by evaluating several reasoning models, including OpenAI’s o3, Anthropic’s Claude Sonnet 4, and Google’s Gemini 2.5 Pro, on 480 tasks from ConceptARC","url":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and","relationship":"Source-stated"},{"text":"Mitchell reports that models approached or exceeded human accuracy with textual inputs but were more prone than humans to use unintended shortcuts.","excerpt":"We found that while o3, Claude, and Gemini approach or exceed human accuracy on these tasks with textual inputs, they are substantially more prone to use unintended “shortcuts” to solve the tasks than do humans.","url":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and","relationship":"Source-stated"},{"text":"Mitchell reports that with visual inputs the models generated accurate grids poorly, while more often producing intended rules than correct grids.","excerpt":"With visual inputs, these models all do quite poorly on generating accurate grids. However, they do generate “correct as intended” rules more often than they generate correct grids","url":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and","relationship":"Source-stated"},{"text":"Mitchell argues that accuracy alone can overestimate abstract reasoning in the textual setting and underestimate it in the visual setting.","excerpt":"evaluations like those done for the ARC Prize, using accuracy alone, may be overestimating abstract reasoning ability of these models in the textual setting. On the other hand, using accuracy alone may be underestimating the ability for abstract reasoning in the visual setting.","url":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and","relationship":"Source-stated"},{"text":"Mitchell says humanlike abstractions matter for handling novelty and for AI systems to understand human intentions and explain reasoning understandably.","excerpt":"These are all things we want AI systems to do in more trustworthy ways.\nMoreover, if we humans are to work together with AI systems, it’s essential to develop AI models that are better at grasping our abstractions","url":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well do these findings about ConceptARC tasks extend to other evaluations of abstract reasoning?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Melanie Mitchell argues that accuracy can misread abstract reasoning","summary":"Melanie Mitchell’s source is dated October 6, 2025. She describes her group’s study of reasoning models on ConceptARC tasks, designed to isolate core concepts. The group compared models’ answers and stated rules. Mitchell reports that with textual inputs, models approached or exceeded human accuracy but used unintended shortcuts more often than humans. With visual inputs, they generated accurate grids poorly, yet produced intended rules more often than correct grids. A rule can fit examples without capturing the abstraction a task is meant to test. Mitchell argues that accuracy alone can misrepresent abstract reasoning, depending on the input type. She says humanlike abstractions help people handle novelty and predict outcomes, and that AI systems need to grasp human intentions and explain their reasoning understandably. Her conclusion is that evaluations need to go beyond accuracy. The account does not establish how well these findings extend beyond the tested tasks and inputs.","question":"How well do these findings about ConceptARC tasks extend to other evaluations of abstract reasoning?","claims":[{"text":"Melanie Mitchell says her group evaluated reasoning models on ConceptARC tasks designed to isolate core concepts.","excerpt":"we investigated this question by evaluating several reasoning models, including OpenAI’s o3, Anthropic’s Claude Sonnet 4, and Google’s Gemini 2.5 Pro, on 480 tasks from ConceptARC","support":"source-stated"},{"text":"Mitchell reports that models approached or exceeded human accuracy with textual inputs but were more prone than humans to use unintended shortcuts.","excerpt":"We found that while o3, Claude, and Gemini approach or exceed human accuracy on these tasks with textual inputs, they are substantially more prone to use unintended “shortcuts” to solve the tasks than do humans.","support":"source-stated"},{"text":"Mitchell reports that with visual inputs the models generated accurate grids poorly, while more often producing intended rules than correct grids.","excerpt":"With visual inputs, these models all do quite poorly on generating accurate grids. However, they do generate “correct as intended” rules more often than they generate correct grids","support":"source-stated"},{"text":"Mitchell argues that accuracy alone can overestimate abstract reasoning in the textual setting and underestimate it in the visual setting.","excerpt":"evaluations like those done for the ARC Prize, using accuracy alone, may be overestimating abstract reasoning ability of these models in the textual setting. On the other hand, using accuracy alone may be underestimating the ability for abstract reasoning in the visual setting.","support":"source-stated"},{"text":"Mitchell says humanlike abstractions matter for handling novelty and for AI systems to understand human intentions and explain reasoning understandably.","excerpt":"These are all things we want AI systems to do in more trustworthy ways.\nMoreover, if we humans are to work together with AI systems, it’s essential to develop AI models that are better at grasping our abstractions","support":"source-stated"}],"limitations":["Text as retrieved on October 23, 2026; originally published October 6, 2025.","The account describes results on ConceptARC tasks and the tested input types; it does not establish how the findings extend to other tasks or real-world settings."],"mediaId":null,"correctionReason":null,"id":"cad9ba4cfc10c0fbd3803e5be856a06ed88c23efb48526adaf419495a84797a8","publicationId":"848f9634-2f85-ba2c-0f0f-e0bdd5a474e6","version":1,"sourceUrl":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and","sourceTitle":"Do AI Reasoning Models Abstract and Reason Like Humans?","publisher":"Melanie Mitchell","publishedAt":"2025-10-06T21:00:25.000Z","originalPublished":{"value":"2025-10-06T21:00:25.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-23T23:46:08.144Z","createdAt":"2026-11-11T20:43:23.872Z","historical":true,"entityIds":[],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://aiguide.substack.com/p/do-ai-reasoning-models-abstract-and","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"88121d7f9cf66b3eb9164118f610cef1c01b8d277a0ed63a3a84bf336017d468","rubricVersion":"unrecorded","provider":"unrecorded","model":"anthropic/claude-opus-5.5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-23T23:46:04.042Z"},"provenance":{"textHash":"55b8086d47eced7cc97ffa494ba6c7622e44b5abea60cb274973b566ef66b0a1","observationId":"ce7a145cacd345185f7ae8b69fcf22a62e248a74c7a0dd7508f41dd2c426358f","retrievedAt":"2026-10-23T23:47:38.287Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"a036b2d1-4db5-77cd-814b-4735fa3b5731"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"f93eb0fc-acf2-c931-aa43-bc6f48a45dee"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-11T20:40:01.988Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"848f9634-2f85-ba2c-0f0f-e0bdd5a474e6","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the post and the study to Melanie Mitchell and her group. This matches the byline and her statement that she is summarizing a new paper from her group. It is one authored piece, not a listing.","excerpt":"Here I’ll summarize a new paper from my group"},{"dimension":"summary","status":"uncertain","explanation":"The summary is faithful, but a non-specialist cannot tell what the tasks are. It refers to models generating \"accurate grids\" and \"correct grids\" without explaining that ConceptARC tasks show example input grids and ask for the output grid. It also does not say that textual inputs are grids written as number matrices while visual inputs are images. Smallest repair: one sentence saying tasks show example colored-grid transformations and ask for the output grid, given as numbers (text) or images (visual). The phrase \"depending on the input type\" also blurs the direction of the effect: overestimation for text, underestimation for visuals.","excerpt":"The task gives three demonstrations of a transformation rule that maps an input grid to an output grid."},{"dimension":"date","status":"pass","explanation":"October 6, 2025 matches the dateline and the UTC day of originalPublished. The modified date equals the publication date. Nothing is presented as known after that date, and the page is a publisher page, so no archive disclosure is needed.","excerpt":"Melanie Mitchell\nOct 06, 2025"},{"dimension":"support","status":"pass","explanation":"Each claim matches its quoted source text: textual-input accuracy combined with more shortcuts, poor visual grids alongside more intended rules than correct grids, and the over- and under-estimation argument. The reasons why abstractions matter are also supported. The limitation and the question are framed as open, not asserted.","excerpt":"they are substantially more prone to use unintended “shortcuts” to solve the tasks than do humans"},{"dimension":"media","status":"pass","explanation":"No media was selected, and no verified candidates were supplied, so nothing is misidentified.","excerpt":null}],"createdAt":"2026-11-11T20:40:01.988Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}