# Melanie Mitchell argues counterfactual tasks can test LLM reasoning

[Canonical reading](https://escapemeridian.com/readings/melanie-mitchell-argues-counterfactual-tasks-can-test-llm-5a3bb9c2)

Melanie Mitchell’s post was published May 13, 2024. She argues that strong performance on familiar tests may reflect patterns in training data rather than general reasoning. Counterfactual tasks compare tasks requiring the same kinds of reasoning but change the content to make it less likely to resemble training examples. In a chess example, GPT-4’s accuracy fell to about 54% when asked to apply chess rules after knights and bishops swapped starting positions. Mitchell also describes word-reversal tests where GPT-3.5 and GPT-4 did worse when the correct output was unlikely under their token probabilities. She says this suggests the models’ training objective—predicting the next token—can affect performance even when that probability is irrelevant to the task. Her conclusion is that LLMs seem to have some ability to reason, but without stress-testing, their performance does not establish general reasoning that will transfer to unfamiliar examples. Mitchell says her group is applying the approach to claims about analogical reasoning. The post leaves open how well these tests predict performance in practical settings.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

Melanie Mitchell · Article

Original source date: May 13, 2024 (2024-05-13T23:41:40.000Z).

Date basis: Date supplied by the publisher’s feed.

Added to Escape Meridian: 2026-11-18T16:48:09.528Z. First observed: 2026-10-24T02:36:11.951Z. Retrieved: 2026-11-05T11:35:37.642Z.

- [Evaluating Large Language Models Using “Counterfactual Tasks”](https://aiguide.substack.com/p/evaluating-large-language-models)

## Limits

Text as retrieved on November 5, 2026; originally published May 13, 2024. The post describes evaluation examples and conclusions, but does not establish how well these tests predict performance in practical settings.


## Open question

How well do results on counterfactual tasks predict whether LLMs can reason reliably in practical settings?


## Source-stated

Mitchell says benchmark results may not show whether LLMs reason generally or rely on patterns in training data.

> Are these models understanding and reasoning in a general way, or are they doing what AI researcher Subbarao Kambhampati calls “approximate retrieval”

- [Supporting source](https://aiguide.substack.com/p/evaluating-large-language-models)

## Source-stated

In the chess example, GPT-4’s accuracy fell from nearly 90% on the familiar task to about 54% on a counterfactual version.

> GPT-4’s accuracy on this counterfactual task dropped to about 54% (random guessing would yield 50%).

- [Supporting source](https://aiguide.substack.com/p/evaluating-large-language-models)

## Source-stated

Mitchell reports that GPT-3.5 and GPT-4 did worse at word reversal when the correct output sentence was unlikely under their computed token probabilities.

> these systems were much worse at the task when the output sentence was unlikely

- [Supporting source](https://aiguide.substack.com/p/evaluating-large-language-models)

## Source-stated

Mitchell concludes that without stress-testing, performance cannot establish general reasoning rather than reliance on training data that may not generalize.

> without stress-testing them (e.g., with counterfactual tasks), one can’t conclude that they are reasoning in a general way rather than relying on their training data in ways that won’t generalize to out-of-distribution examples.

- [Supporting source](https://aiguide.substack.com/p/evaluating-large-language-models)

## Source-stated

Mitchell says her group plans to apply the paradigm to stress-test claims about analogical reasoning.

> I’ll describe work my group is doing on applying the counterfactual-task paradigm to stress-test claims that LLMs are robust and general analogical reasoners.

- [Supporting source](https://aiguide.substack.com/p/evaluating-large-language-models)

## Source and contributor context

Melanie Mitchell · Publisher

The publisher is identified here. No individual author has been inferred.


## Source media

Media retrieval status: unchecked.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Counterfactual

An argument or scenario that considers what would have resulted if events had gone differently from how they actually did, such as if a policy had never been adopted; as an adjective, relating to this kind of thinking about what did not happen but could have.

- [Definition reference](https://dictionary.cambridge.org/dictionary/english/counterfactual)

## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## Editorial provenance

Version 1. Publication ID: c4baaa3b-8431-5f6b-11ac-b17c3c94f1a7.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T16:43:31.731Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge\_admitted\_by\_model, second\_opinion\_categorical.

Observation: 815dc3b777f0a558663af055d837d670c58505bca177c25d54e5cbf1d7689bde. Source text hash: acc24653a41ca7bf870a666f4f450d8a3d7c3e2ad1817e896129228a9b708356.


## Revision and source-check history

### Version 1 — 2026-11-18T16:48:09.528Z

Melanie Mitchell’s post was published May 13, 2024. She argues that strong performance on familiar tests may reflect patterns in training data rather than general reasoning. Counterfactual tasks compare tasks requiring the same kinds of reasoning but change the content to make it less likely to resemble training examples. In a chess example, GPT-4’s accuracy fell to about 54% when asked to apply chess rules after knights and bishops swapped starting positions. Mitchell also describes word-reversal tests where GPT-3.5 and GPT-4 did worse when the correct output was unlikely under their token probabilities. She says this suggests the models’ training objective—predicting the next token—can affect performance even when that probability is irrelevant to the task. Her conclusion is that LLMs seem to have some ability to reason, but without stress-testing, their performance does not establish general reasoning that will transfer to unfamiliar examples. Mitchell says her group is applying the approach to claims about analogical reasoning. The post leaves open how well these tests predict performance in practical settings.

## Original source

Melanie Mitchell · Article

Original source date: May 13, 2024 (2024-05-13T23:41:40.000Z).

Date basis: Date supplied by the publisher’s feed.

Added to Escape Meridian: 2026-11-18T16:48:09.528Z. First observed: 2026-10-24T02:36:11.951Z. Retrieved: 2026-11-05T11:35:37.642Z.

- [Evaluating Large Language Models Using “Counterfactual Tasks”](https://aiguide.substack.com/p/evaluating-large-language-models)

## Limits

Text as retrieved on November 5, 2026; originally published May 13, 2024. The post describes evaluation examples and conclusions, but does not establish how well these tests predict performance in practical settings.


## Open question

How well do results on counterfactual tasks predict whether LLMs can reason reliably in practical settings?


## Source-stated

Mitchell says benchmark results may not show whether LLMs reason generally or rely on patterns in training data.

> Are these models understanding and reasoning in a general way, or are they doing what AI researcher Subbarao Kambhampati calls “approximate retrieval”

- [Supporting source](https://aiguide.substack.com/p/evaluating-large-language-models)

## Source-stated

In the chess example, GPT-4’s accuracy fell from nearly 90% on the familiar task to about 54% on a counterfactual version.

> GPT-4’s accuracy on this counterfactual task dropped to about 54% (random guessing would yield 50%).

- [Supporting source](https://aiguide.substack.com/p/evaluating-large-language-models)

## Source-stated

Mitchell reports that GPT-3.5 and GPT-4 did worse at word reversal when the correct output sentence was unlikely under their computed token probabilities.

> these systems were much worse at the task when the output sentence was unlikely

- [Supporting source](https://aiguide.substack.com/p/evaluating-large-language-models)

## Source-stated

Mitchell concludes that without stress-testing, performance cannot establish general reasoning rather than reliance on training data that may not generalize.

> without stress-testing them (e.g., with counterfactual tasks), one can’t conclude that they are reasoning in a general way rather than relying on their training data in ways that won’t generalize to out-of-distribution examples.

- [Supporting source](https://aiguide.substack.com/p/evaluating-large-language-models)

## Source-stated

Mitchell says her group plans to apply the paradigm to stress-test claims about analogical reasoning.

> I’ll describe work my group is doing on applying the counterfactual-task paradigm to stress-test claims that LLMs are robust and general analogical reasoners.

- [Supporting source](https://aiguide.substack.com/p/evaluating-large-language-models)

## Source and contributor context

Melanie Mitchell · Publisher

The publisher is identified here. No individual author has been inferred.


## Source media

Media retrieval status: unchecked.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Counterfactual

An argument or scenario that considers what would have resulted if events had gone differently from how they actually did, such as if a policy had never been adopted; as an adjective, relating to this kind of thinking about what did not happen but could have.

- [Definition reference](https://dictionary.cambridge.org/dictionary/english/counterfactual)

## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## Editorial provenance

Version 1. Publication ID: c4baaa3b-8431-5f6b-11ac-b17c3c94f1a7.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T16:43:31.731Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge\_admitted\_by\_model, second\_opinion\_categorical.

Observation: 815dc3b777f0a558663af055d837d670c58505bca177c25d54e5cbf1d7689bde. Source text hash: acc24653a41ca7bf870a666f4f450d8a3d7c3e2ad1817e896129228a9b708356.


### uncertain — 2026-11-18T16:43:31.731Z — anthropic/claude-opus-5.5

Publication: c4baaa3b-8431-5f6b-11ac-b17c3c94f1a7

attribution — uncertain: The post and its argument are correctly attributed to Melanie Mitchell. However, the chess and word-reversal results come from papers by other researchers that she summarizes: “Reasoning or Reciting” and “Embers of Autoregression.” The reading presents them only as “a chess example” and tests Mitchell “describes,” so a reader could take them for her own experiments. Smallest repair: say these results come from papers by other researchers that Mitchell cites.

> For example, the paper “Reasoning or Reciting: Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks” looked at GPT-4’s performance

summary — pass: The reading leads with Mitchell’s argument and explains counterfactual tasks in plain language. It gives both concrete examples and preserves her hedged conclusion that LLMs seem to have some ability to reason, but that general reasoning can't be concluded without stress-testing. The question follows from the evidence. LLM is defined beside the reading. Next-token prediction is glossed in context.

> In short, LLMs seem to have some ability to reason, but without stress-testing them

date — pass: The stated date, May 13, 2024, matches the UTC day of originalPublished and the post’s dateline. This is a publisher page, not an archive copy. The reading uses no later events or hindsight framing. Mitchell’s announced next post is reported as her group’s ongoing work, not as an outcome.

> Melanie Mitchell
May 13, 2024

support — pass: Each claim is supported by the source:
- The figures nearly 90%, about 54% and the 50% chance baseline.
- The word-reversal finding.
- The training-objective interpretation.
- Her concluding caveat.
Claim 5 says the group “plans” to apply the approach, while the source says the work is under way. This is a minor nuance; the summary’s “is applying” matches the source. The stated limitation is framed as an open question, not a claim.

> GPT-4’s accuracy on this counterfactual task dropped to about 54% (random guessing would yield 50%).

media — pass: No media is selected, and no verified media candidates were supplied, so no media was required.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/5a3bb9c2b4dc0ca6a0657843be3cd4bff14d3ef2629b0a1119c30ee27366fdc0/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/melanie-mitchell-argues-counterfactual-tasks-can-test-llm-5a3bb9c2.json)
