Escape Meridian

Melanie Mitchell · May 21, 2024

Melanie Mitchell argues that analogy tests should probe abstract reasoning

Markdown · JSON · Original source

Melanie Mitchell’s post, published May 21, 2024, examines a dispute over whether can reason by analogy. Mitchell and collaborator Martha Lewis report that their replication found humans outperformed GPT-3, GPT-3.5 and GPT-4 on the simplest letter-string problems; Mitchell says the difference from an earlier study remains unexplained. In tests using fictional alphabets and symbols, she reports that human accuracy stayed relatively high while model accuracy dropped substantially, even after the models showed they understood the fictional alphabet’s order. Taylor Webb and coauthors argued that solving permuted-alphabet problems required converting letters to positions. A GPT-4 version able to generate code used that approach and performed above humans on several types of problems. Mitchell disputes that this tests the same abstract reasoning: she argues people can use successorship and predecessorship without counting positions. That distinction matters because the tasks were designed to probe general analogy, not arithmetic about letters. Mitchell closes by emphasizing stress-testing, replication and scientific debate. The studies leave unresolved what the models’ results say about abstract reasoning, and whether code-assisted solving tests the ability the tasks were intended to assess.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

Melanie MitchellArticle · aiguide.substack.comOpen original source

Original source

Melanie Mitchell · Article

Source details

Original source date: May 21, 2024 (2024-05-21T17:42:04.000Z).

Date basis: Date supplied by the publisher’s feed.

Added to Escape Meridian: 2026-11-18T15:08:09.982Z. First observed: 2026-10-24T01:52:00.948Z. Retrieved: 2026-10-24T02:09:38.316Z.

Referenced by

Recorded citations in public readings. A reference alone does not establish agreement or influence.

No public references recorded here yet.

Reference data (JSON)

Limits

Text as retrieved on October 24, 2026; originally published May 21, 2024. Mitchell says the differing results between her replication and Webb and coauthors’ study are unexplained, and suggests the participant populations may have contributed. The post does not resolve the disagreement over whether index-based code solving demonstrates the abstract reasoning the tasks were intended to test.

Open question

Do the code-assisted results test the same abstract understanding of successorship and predecessorship that Mitchell says the tasks were designed to assess?

Source-stated

Mitchell reports that her and Lewis’s replication found humans outperformed GPT-3, GPT-3.5 and GPT-4 on the simplest letter-string analogy problems; she says the difference from the earlier study remains unexplained.

on average, on the zero-generalization problems, humans outperformed GPT-3 as well as more recent versions, GPT-3.5 and GPT-4. We don’t know what accounts for the difference from Webb et al.’s results

Source-stated

Mitchell reports that model accuracy fell substantially on counterfactual tasks using fictional alphabets, while human accuracy stayed relatively high.

the accuracy of humans stays relatively high, while the accuracy of the LLMs drops substantially.

Source-stated

Webb and coauthors argued that solving permuted-alphabet problems required converting letters to indices; Mitchell rejects that as an account of how humans solve them and as a test of the intended abstract reasoning.

I disagree that the letter-string problems with permuted alphabets “require that letters be converted into the corresponding indices.”

Source-stated

Mitchell argues that replication and counterfactual tasks matter for evaluating reasoning claims about language models, and calls for scientific debate.

I hope I’ve communicated the importance of doing such experiments, as well as of scientific replication studies. And, of course, of healthy scientific debates!

Source and contributor context

Melanie Mitchell · Publisher

The publisher is identified here. No individual author has been inferred.

Source media

Media retrieval status: unchecked.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Large language model

A language model with many learned numerical settings, used to process and generate language.

Counterfactual

An argument or scenario that considers what would have resulted if events had gone differently from how they actually did, such as if a policy had never been adopted; as an adjective, relating to this kind of thinking about what did not happen but could have.

Editorial provenance

Version 1. Publication ID: 4dde8247-ad99-d3b7-4ebc-46833c0861bb.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T15:03:34.564Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge_admitted_by_model, second_opinion_categorical.

Observation: 473306279a40a6e46d4422a9cf6e739b07cf8b5351a6c7f0a5c43b20a1e46822. Source text hash: 009a8acbc710627d052d5b1e8460e18ddf3151e3790c3da50da996b7aa32877c.

Revision and source-check history

Version 1 · 2026-11-18T15:08:09.982Z

Melanie Mitchell argues that analogy tests should probe abstract reasoning

Melanie Mitchell’s post, published May 21, 2024, examines a dispute over whether can reason by analogy. Mitchell and collaborator Martha Lewis report that their replication found humans outperformed GPT-3, GPT-3.5 and GPT-4 on the simplest letter-string problems; Mitchell says the difference from an earlier study remains unexplained. In tests using fictional alphabets and symbols, she reports that human accuracy stayed relatively high while model accuracy dropped substantially, even after the models showed they understood the fictional alphabet’s order. Taylor Webb and coauthors argued that solving permuted-alphabet problems required converting letters to positions. A GPT-4 version able to generate code used that approach and performed above humans on several types of problems. Mitchell disputes that this tests the same abstract reasoning: she argues people can use successorship and predecessorship without counting positions. That distinction matters because the tasks were designed to probe general analogy, not arithmetic about letters. Mitchell closes by emphasizing stress-testing, replication and scientific debate. The studies leave unresolved what the models’ results say about abstract reasoning, and whether code-assisted solving tests the ability the tasks were intended to assess.

Original source

Melanie Mitchell · Article

Source details

Original source date: May 21, 2024 (2024-05-21T17:42:04.000Z).

Date basis: Date supplied by the publisher’s feed.

Added to Escape Meridian: 2026-11-18T15:08:09.982Z. First observed: 2026-10-24T01:52:00.948Z. Retrieved: 2026-10-24T02:09:38.316Z.

Limits

Text as retrieved on October 24, 2026; originally published May 21, 2024. Mitchell says the differing results between her replication and Webb and coauthors’ study are unexplained, and suggests the participant populations may have contributed. The post does not resolve the disagreement over whether index-based code solving demonstrates the abstract reasoning the tasks were intended to test.

Open question

Do the code-assisted results test the same abstract understanding of successorship and predecessorship that Mitchell says the tasks were designed to assess?

Source-stated

Mitchell reports that her and Lewis’s replication found humans outperformed GPT-3, GPT-3.5 and GPT-4 on the simplest letter-string analogy problems; she says the difference from the earlier study remains unexplained.

on average, on the zero-generalization problems, humans outperformed GPT-3 as well as more recent versions, GPT-3.5 and GPT-4. We don’t know what accounts for the difference from Webb et al.’s results

Source-stated

Mitchell reports that model accuracy fell substantially on counterfactual tasks using fictional alphabets, while human accuracy stayed relatively high.

the accuracy of humans stays relatively high, while the accuracy of the LLMs drops substantially.

Source-stated

Webb and coauthors argued that solving permuted-alphabet problems required converting letters to indices; Mitchell rejects that as an account of how humans solve them and as a test of the intended abstract reasoning.

I disagree that the letter-string problems with permuted alphabets “require that letters be converted into the corresponding indices.”

Source-stated

Mitchell argues that replication and counterfactual tasks matter for evaluating reasoning claims about language models, and calls for scientific debate.

I hope I’ve communicated the importance of doing such experiments, as well as of scientific replication studies. And, of course, of healthy scientific debates!

Source and contributor context

Melanie Mitchell · Publisher

The publisher is identified here. No individual author has been inferred.

Source media

Media retrieval status: unchecked.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Large language model

A language model with many learned numerical settings, used to process and generate language.

Counterfactual

An argument or scenario that considers what would have resulted if events had gone differently from how they actually did, such as if a policy had never been adopted; as an adjective, relating to this kind of thinking about what did not happen but could have.

Editorial provenance

Version 1. Publication ID: 4dde8247-ad99-d3b7-4ebc-46833c0861bb.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T15:03:34.564Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge_admitted_by_model, second_opinion_categorical.

Observation: 473306279a40a6e46d4422a9cf6e739b07cf8b5351a6c7f0a5c43b20a1e46822. Source text hash: 009a8acbc710627d052d5b1e8460e18ddf3151e3790c3da50da996b7aa32877c.

uncertain · 2026-11-18T15:03:34.564Z · anthropic/claude-opus-5.5

Publication: 4dde8247-ad99-d3b7-4ebc-46833c0861bb

attribution — pass: This is a single Substack post by Melanie Mitchell. The reading attributes the findings to Mitchell and Lewis, attributes the counter-argument to Webb and coauthors, and keeps Mitchell's disagreement framed as her own argument.

Webb et al. did not agree with our assessment.

summary — uncertain: Two gaps limit a general reader's understanding. First, 'letter-string problems' is never explained, for example with abc to abcd. Second, the earlier study is never described: Webb et al. found GPT-3 beat UCLA undergraduates, which is the claim Mitchell's replication tests. Taylor Webb also appears later without being identified as that study's author. Smallest repair: one clause with an example problem, and one clause stating that Webb et al. originally reported GPT-3 matching or surpassing humans.

they found that GPT-3 was more accurate (had more correct answers) than the average of a group of UCLA undergraduates the researchers tested.

date — pass: The originalPublished instant 2024-05-21T17:42:04Z matches the dateline May 21, 2024, and the reading states it at day precision. This is a publisher page, not an archive copy. The reading adds no hindsight, and modifiedAt equals the publication time.

May 21, 2024

support — uncertain: Most claims match the source. One phrase drops Mitchell's hedging: 'even after the models showed they understood the fictional alphabet's order.' Mitchell puts understood in scare quotes, defines it narrowly as naming the next and previous letter, and says the models 'generally' did well. Smallest repair: 'even after checking that the models could generally name each letter's neighbours in the fictional alphabet.'

we made sure that the LLMs “understood” the fictional alphabet, meaning that they could say what the “next letter” and “previous letter” was of each letter in the given fictional alphabet—the LLMs generally did very well on this.

media — pass: No media was selected and no verified candidates were supplied, so there is nothing to check for identity or relevance.

Challenge or correct this reading

Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.