{"id":"5a3bb9c2b4dc0ca6a0657843be3cd4bff14d3ef2629b0a1119c30ee27366fdc0","title":"Melanie Mitchell argues counterfactual tasks can test LLM reasoning","argument":"Melanie Mitchell’s post was published May 13, 2024. She argues that strong performance on familiar tests may reflect patterns in training data rather than general reasoning. Counterfactual tasks compare tasks requiring the same kinds of reasoning but change the content to make it less likely to resemble training examples. In a chess example, GPT-4’s accuracy fell to about 54% when asked to apply chess rules after knights and bishops swapped starting positions. Mitchell also describes word-reversal tests where GPT-3.5 and GPT-4 did worse when the correct output was unlikely under their token probabilities. She says this suggests the models’ training objective—predicting the next token—can affect performance even when that probability is irrelevant to the task. Her conclusion is that LLMs seem to have some ability to reason, but without stress-testing, their performance does not establish general reasoning that will transfer to unfamiliar examples. Mitchell says her group is applying the approach to claims about analogical reasoning. The post leaves open how well these tests predict performance in practical settings.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on November 5, 2026; originally published May 13, 2024. The post describes evaluation examples and conclusions, but does not establish how well these tests predict performance in practical settings.","speaker":{"name":"Melanie Mitchell","roleAtSource":"Publisher"},"source":{"title":"Evaluating Large Language Models Using “Counterfactual Tasks”","publisher":"Melanie Mitchell","format":"Article","url":"https://aiguide.substack.com/p/evaluating-large-language-models","publishedAt":"2024-05-13T23:41:40.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"Mitchell says benchmark results may not show whether LLMs reason generally or rely on patterns in training data.","excerpt":"Are these models understanding and reasoning in a general way, or are they doing what AI researcher Subbarao Kambhampati calls “approximate retrieval”","url":"https://aiguide.substack.com/p/evaluating-large-language-models","relationship":"Source-stated"},{"text":"In the chess example, GPT-4’s accuracy fell from nearly 90% on the familiar task to about 54% on a counterfactual version.","excerpt":"GPT-4’s accuracy on this counterfactual task dropped to about 54% (random guessing would yield 50%).","url":"https://aiguide.substack.com/p/evaluating-large-language-models","relationship":"Source-stated"},{"text":"Mitchell reports that GPT-3.5 and GPT-4 did worse at word reversal when the correct output sentence was unlikely under their computed token probabilities.","excerpt":"these systems were much worse at the task when the output sentence was unlikely","url":"https://aiguide.substack.com/p/evaluating-large-language-models","relationship":"Source-stated"},{"text":"Mitchell concludes that without stress-testing, performance cannot establish general reasoning rather than reliance on training data that may not generalize.","excerpt":"without stress-testing them (e.g., with counterfactual tasks), one can’t conclude that they are reasoning in a general way rather than relying on their training data in ways that won’t generalize to out-of-distribution examples.","url":"https://aiguide.substack.com/p/evaluating-large-language-models","relationship":"Source-stated"},{"text":"Mitchell says her group plans to apply the paradigm to stress-test claims about analogical reasoning.","excerpt":"I’ll describe work my group is doing on applying the counterfactual-task paradigm to stress-test claims that LLMs are robust and general analogical reasoners.","url":"https://aiguide.substack.com/p/evaluating-large-language-models","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://aiguide.substack.com/p/evaluating-large-language-models","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well do results on counterfactual tasks predict whether LLMs can reason reliably in practical settings?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Melanie Mitchell argues counterfactual tasks can test LLM reasoning","summary":"Melanie Mitchell’s post was published May 13, 2024. She argues that strong performance on familiar tests may reflect patterns in training data rather than general reasoning. Counterfactual tasks compare tasks requiring the same kinds of reasoning but change the content to make it less likely to resemble training examples. In a chess example, GPT-4’s accuracy fell to about 54% when asked to apply chess rules after knights and bishops swapped starting positions. Mitchell also describes word-reversal tests where GPT-3.5 and GPT-4 did worse when the correct output was unlikely under their token probabilities. She says this suggests the models’ training objective—predicting the next token—can affect performance even when that probability is irrelevant to the task. Her conclusion is that LLMs seem to have some ability to reason, but without stress-testing, their performance does not establish general reasoning that will transfer to unfamiliar examples. Mitchell says her group is applying the approach to claims about analogical reasoning. The post leaves open how well these tests predict performance in practical settings.","question":"How well do results on counterfactual tasks predict whether LLMs can reason reliably in practical settings?","claims":[{"text":"Mitchell says benchmark results may not show whether LLMs reason generally or rely on patterns in training data.","excerpt":"Are these models understanding and reasoning in a general way, or are they doing what AI researcher Subbarao Kambhampati calls “approximate retrieval”","support":"source-stated"},{"text":"In the chess example, GPT-4’s accuracy fell from nearly 90% on the familiar task to about 54% on a counterfactual version.","excerpt":"GPT-4’s accuracy on this counterfactual task dropped to about 54% (random guessing would yield 50%).","support":"source-stated"},{"text":"Mitchell reports that GPT-3.5 and GPT-4 did worse at word reversal when the correct output sentence was unlikely under their computed token probabilities.","excerpt":"these systems were much worse at the task when the output sentence was unlikely","support":"source-stated"},{"text":"Mitchell concludes that without stress-testing, performance cannot establish general reasoning rather than reliance on training data that may not generalize.","excerpt":"without stress-testing them (e.g., with counterfactual tasks), one can’t conclude that they are reasoning in a general way rather than relying on their training data in ways that won’t generalize to out-of-distribution examples.","support":"source-stated"},{"text":"Mitchell says her group plans to apply the paradigm to stress-test claims about analogical reasoning.","excerpt":"I’ll describe work my group is doing on applying the counterfactual-task paradigm to stress-test claims that LLMs are robust and general analogical reasoners.","support":"source-stated"}],"limitations":["Text as retrieved on November 5, 2026; originally published May 13, 2024.","The post describes evaluation examples and conclusions, but does not establish how well these tests predict performance in practical settings."],"mediaId":null,"correctionReason":null,"id":"5a3bb9c2b4dc0ca6a0657843be3cd4bff14d3ef2629b0a1119c30ee27366fdc0","publicationId":"c4baaa3b-8431-5f6b-11ac-b17c3c94f1a7","version":1,"sourceUrl":"https://aiguide.substack.com/p/evaluating-large-language-models","sourceTitle":"Evaluating Large Language Models Using “Counterfactual Tasks”","publisher":"Melanie Mitchell","publishedAt":"2024-05-13T23:41:40.000Z","originalPublished":{"value":"2024-05-13T23:41:40.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-24T02:36:11.951Z","createdAt":"2026-11-18T16:48:09.528Z","historical":true,"entityIds":[],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://aiguide.substack.com/p/evaluating-large-language-models","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"998da2fa381db0bec27f7125ebe6208a176ea166c1417e8411da8c194ab98add","rubricVersion":"unrecorded","provider":"unrecorded","model":"anthropic/claude-opus-5.5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-24T02:36:08.033Z"},"provenance":{"textHash":"acc24653a41ca7bf870a666f4f450d8a3d7c3e2ad1817e896129228a9b708356","observationId":"815dc3b777f0a558663af055d837d670c58505bca177c25d54e5cbf1d7689bde","retrievedAt":"2026-11-05T11:35:37.642Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"0648c499-7201-b27e-8c0b-fba9cfce60a0"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"0de9f2f3-026d-6924-bde6-100fd268d66d"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-18T16:43:31.731Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/melanie-mitchell-argues-counterfactual-tasks-can-test-llm-5a3bb9c2","formats":{"html":"/readings/melanie-mitchell-argues-counterfactual-tasks-can-test-llm-5a3bb9c2","markdown":"/readings/melanie-mitchell-argues-counterfactual-tasks-can-test-llm-5a3bb9c2.md","json":"/readings/melanie-mitchell-argues-counterfactual-tasks-can-test-llm-5a3bb9c2.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-counterfactual","name":"Counterfactual","explanation":"An argument or scenario that considers what would have resulted if events had gone differently from how they actually did, such as if a policy had never been adopted; as an adjective, relating to this kind of thinking about what did not happen but could have.","referenceUrl":"https://dictionary.cambridge.org/dictionary/english/counterfactual"},{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"}],"entities":[]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"5a3bb9c2b4dc0ca6a0657843be3cd4bff14d3ef2629b0a1119c30ee27366fdc0","revision":"c4baaa3b-8431-5f6b-11ac-b17c3c94f1a7","revisionKind":"publication-id","title":"Melanie Mitchell argues counterfactual tasks can test LLM reasoning","url":"https://escapemeridian.com/readings/melanie-mitchell-argues-counterfactual-tasks-can-test-llm-5a3bb9c2"},"source":{"type":"source","id":"https://aiguide.substack.com/p/evaluating-large-language-models","revision":"acc24653a41ca7bf870a666f4f450d8a3d7c3e2ad1817e896129228a9b708356","revisionKind":"sha256-source-text","title":"Evaluating Large Language Models Using “Counterfactual Tasks”","url":"https://aiguide.substack.com/p/evaluating-large-language-models"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"5a3bb9c2b4dc0ca6a0657843be3cd4bff14d3ef2629b0a1119c30ee27366fdc0","title":"Melanie Mitchell argues counterfactual tasks can test LLM reasoning","argument":"Melanie Mitchell’s post was published May 13, 2024. She argues that strong performance on familiar tests may reflect patterns in training data rather than general reasoning. Counterfactual tasks compare tasks requiring the same kinds of reasoning but change the content to make it less likely to resemble training examples. In a chess example, GPT-4’s accuracy fell to about 54% when asked to apply chess rules after knights and bishops swapped starting positions. Mitchell also describes word-reversal tests where GPT-3.5 and GPT-4 did worse when the correct output was unlikely under their token probabilities. She says this suggests the models’ training objective—predicting the next token—can affect performance even when that probability is irrelevant to the task. Her conclusion is that LLMs seem to have some ability to reason, but without stress-testing, their performance does not establish general reasoning that will transfer to unfamiliar examples. Mitchell says her group is applying the approach to claims about analogical reasoning. The post leaves open how well these tests predict performance in practical settings.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on November 5, 2026; originally published May 13, 2024. The post describes evaluation examples and conclusions, but does not establish how well these tests predict performance in practical settings.","speaker":{"name":"Melanie Mitchell","roleAtSource":"Publisher"},"source":{"title":"Evaluating Large Language Models Using “Counterfactual Tasks”","publisher":"Melanie Mitchell","format":"Article","url":"https://aiguide.substack.com/p/evaluating-large-language-models","publishedAt":"2024-05-13T23:41:40.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"Mitchell says benchmark results may not show whether LLMs reason generally or rely on patterns in training data.","excerpt":"Are these models understanding and reasoning in a general way, or are they doing what AI researcher Subbarao Kambhampati calls “approximate retrieval”","url":"https://aiguide.substack.com/p/evaluating-large-language-models","relationship":"Source-stated"},{"text":"In the chess example, GPT-4’s accuracy fell from nearly 90% on the familiar task to about 54% on a counterfactual version.","excerpt":"GPT-4’s accuracy on this counterfactual task dropped to about 54% (random guessing would yield 50%).","url":"https://aiguide.substack.com/p/evaluating-large-language-models","relationship":"Source-stated"},{"text":"Mitchell reports that GPT-3.5 and GPT-4 did worse at word reversal when the correct output sentence was unlikely under their computed token probabilities.","excerpt":"these systems were much worse at the task when the output sentence was unlikely","url":"https://aiguide.substack.com/p/evaluating-large-language-models","relationship":"Source-stated"},{"text":"Mitchell concludes that without stress-testing, performance cannot establish general reasoning rather than reliance on training data that may not generalize.","excerpt":"without stress-testing them (e.g., with counterfactual tasks), one can’t conclude that they are reasoning in a general way rather than relying on their training data in ways that won’t generalize to out-of-distribution examples.","url":"https://aiguide.substack.com/p/evaluating-large-language-models","relationship":"Source-stated"},{"text":"Mitchell says her group plans to apply the paradigm to stress-test claims about analogical reasoning.","excerpt":"I’ll describe work my group is doing on applying the counterfactual-task paradigm to stress-test claims that LLMs are robust and general analogical reasoners.","url":"https://aiguide.substack.com/p/evaluating-large-language-models","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://aiguide.substack.com/p/evaluating-large-language-models","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well do results on counterfactual tasks predict whether LLMs can reason reliably in practical settings?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Melanie Mitchell argues counterfactual tasks can test LLM reasoning","summary":"Melanie Mitchell’s post was published May 13, 2024. She argues that strong performance on familiar tests may reflect patterns in training data rather than general reasoning. Counterfactual tasks compare tasks requiring the same kinds of reasoning but change the content to make it less likely to resemble training examples. In a chess example, GPT-4’s accuracy fell to about 54% when asked to apply chess rules after knights and bishops swapped starting positions. Mitchell also describes word-reversal tests where GPT-3.5 and GPT-4 did worse when the correct output was unlikely under their token probabilities. She says this suggests the models’ training objective—predicting the next token—can affect performance even when that probability is irrelevant to the task. Her conclusion is that LLMs seem to have some ability to reason, but without stress-testing, their performance does not establish general reasoning that will transfer to unfamiliar examples. Mitchell says her group is applying the approach to claims about analogical reasoning. The post leaves open how well these tests predict performance in practical settings.","question":"How well do results on counterfactual tasks predict whether LLMs can reason reliably in practical settings?","claims":[{"text":"Mitchell says benchmark results may not show whether LLMs reason generally or rely on patterns in training data.","excerpt":"Are these models understanding and reasoning in a general way, or are they doing what AI researcher Subbarao Kambhampati calls “approximate retrieval”","support":"source-stated"},{"text":"In the chess example, GPT-4’s accuracy fell from nearly 90% on the familiar task to about 54% on a counterfactual version.","excerpt":"GPT-4’s accuracy on this counterfactual task dropped to about 54% (random guessing would yield 50%).","support":"source-stated"},{"text":"Mitchell reports that GPT-3.5 and GPT-4 did worse at word reversal when the correct output sentence was unlikely under their computed token probabilities.","excerpt":"these systems were much worse at the task when the output sentence was unlikely","support":"source-stated"},{"text":"Mitchell concludes that without stress-testing, performance cannot establish general reasoning rather than reliance on training data that may not generalize.","excerpt":"without stress-testing them (e.g., with counterfactual tasks), one can’t conclude that they are reasoning in a general way rather than relying on their training data in ways that won’t generalize to out-of-distribution examples.","support":"source-stated"},{"text":"Mitchell says her group plans to apply the paradigm to stress-test claims about analogical reasoning.","excerpt":"I’ll describe work my group is doing on applying the counterfactual-task paradigm to stress-test claims that LLMs are robust and general analogical reasoners.","support":"source-stated"}],"limitations":["Text as retrieved on November 5, 2026; originally published May 13, 2024.","The post describes evaluation examples and conclusions, but does not establish how well these tests predict performance in practical settings."],"mediaId":null,"correctionReason":null,"id":"5a3bb9c2b4dc0ca6a0657843be3cd4bff14d3ef2629b0a1119c30ee27366fdc0","publicationId":"c4baaa3b-8431-5f6b-11ac-b17c3c94f1a7","version":1,"sourceUrl":"https://aiguide.substack.com/p/evaluating-large-language-models","sourceTitle":"Evaluating Large Language Models Using “Counterfactual Tasks”","publisher":"Melanie Mitchell","publishedAt":"2024-05-13T23:41:40.000Z","originalPublished":{"value":"2024-05-13T23:41:40.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-24T02:36:11.951Z","createdAt":"2026-11-18T16:48:09.528Z","historical":true,"entityIds":[],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://aiguide.substack.com/p/evaluating-large-language-models","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"998da2fa381db0bec27f7125ebe6208a176ea166c1417e8411da8c194ab98add","rubricVersion":"unrecorded","provider":"unrecorded","model":"anthropic/claude-opus-5.5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-24T02:36:08.033Z"},"provenance":{"textHash":"acc24653a41ca7bf870a666f4f450d8a3d7c3e2ad1817e896129228a9b708356","observationId":"815dc3b777f0a558663af055d837d670c58505bca177c25d54e5cbf1d7689bde","retrievedAt":"2026-11-05T11:35:37.642Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"0648c499-7201-b27e-8c0b-fba9cfce60a0"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"0de9f2f3-026d-6924-bde6-100fd268d66d"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-18T16:43:31.731Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"c4baaa3b-8431-5f6b-11ac-b17c3c94f1a7","verdict":"uncertain","findings":[{"dimension":"attribution","status":"uncertain","explanation":"The post and its argument are correctly attributed to Melanie Mitchell. However, the chess and word-reversal results come from papers by other researchers that she summarizes: “Reasoning or Reciting” and “Embers of Autoregression.” The reading presents them only as “a chess example” and tests Mitchell “describes,” so a reader could take them for her own experiments. Smallest repair: say these results come from papers by other researchers that Mitchell cites.","excerpt":"For example, the paper “Reasoning or Reciting: Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks” looked at GPT-4’s performance"},{"dimension":"summary","status":"pass","explanation":"The reading leads with Mitchell’s argument and explains counterfactual tasks in plain language. It gives both concrete examples and preserves her hedged conclusion that LLMs seem to have some ability to reason, but that general reasoning can't be concluded without stress-testing. The question follows from the evidence. LLM is defined beside the reading. Next-token prediction is glossed in context.","excerpt":"In short, LLMs seem to have some ability to reason, but without stress-testing them"},{"dimension":"date","status":"pass","explanation":"The stated date, May 13, 2024, matches the UTC day of originalPublished and the post’s dateline. This is a publisher page, not an archive copy. The reading uses no later events or hindsight framing. Mitchell’s announced next post is reported as her group’s ongoing work, not as an outcome.","excerpt":"Melanie Mitchell\nMay 13, 2024"},{"dimension":"support","status":"pass","explanation":"Each claim is supported by the source:\n- The figures nearly 90%, about 54% and the 50% chance baseline.\n- The word-reversal finding.\n- The training-objective interpretation.\n- Her concluding caveat.\nClaim 5 says the group “plans” to apply the approach, while the source says the work is under way. This is a minor nuance; the summary’s “is applying” matches the source. The stated limitation is framed as an open question, not a claim.","excerpt":"GPT-4’s accuracy on this counterfactual task dropped to about 54% (random guessing would yield 50%)."},{"dimension":"media","status":"pass","explanation":"No media is selected, and no verified media candidates were supplied, so no media was required.","excerpt":null}],"createdAt":"2026-11-18T16:43:31.731Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}