{"id":"36b0c017c513240ded079a838354981d3de1be29cc225428ec3c1a3b83c2bdf0","title":"Melanie Mitchell argues that analogy tests should probe abstract reasoning","argument":"Melanie Mitchell’s post, published May 21, 2024, examines a dispute over whether large language models can reason by analogy. Mitchell and collaborator Martha Lewis report that their replication found humans outperformed GPT-3, GPT-3.5 and GPT-4 on the simplest letter-string problems; Mitchell says the difference from an earlier study remains unexplained. In tests using fictional alphabets and symbols, she reports that human accuracy stayed relatively high while model accuracy dropped substantially, even after the models showed they understood the fictional alphabet’s order.\n\nTaylor Webb and coauthors argued that solving permuted-alphabet problems required converting letters to positions. A GPT-4 version able to generate code used that approach and performed above humans on several types of problems. Mitchell disputes that this tests the same abstract reasoning: she argues people can use successorship and predecessorship without counting positions. That distinction matters because the tasks were designed to probe general analogy, not arithmetic about letters. Mitchell closes by emphasizing stress-testing, replication and scientific debate. The studies leave unresolved what the models’ results say about abstract reasoning, and whether code-assisted solving tests the ability the tasks were intended to assess.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 24, 2026; originally published May 21, 2024. Mitchell says the differing results between her replication and Webb and coauthors’ study are unexplained, and suggests the participant populations may have contributed. The post does not resolve the disagreement over whether index-based code solving demonstrates the abstract reasoning the tasks were intended to test.","speaker":{"name":"Melanie Mitchell","roleAtSource":"Publisher"},"source":{"title":"Stress-Testing Large Language Models’ Analogical Reasoning Abilities","publisher":"Melanie Mitchell","format":"Article","url":"https://aiguide.substack.com/p/stress-testing-large-language-models","publishedAt":"2024-05-21T17:42:04.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"Mitchell reports that her and Lewis’s replication found humans outperformed GPT-3, GPT-3.5 and GPT-4 on the simplest letter-string analogy problems; she says the difference from the earlier study remains unexplained.","excerpt":"on average, on the zero-generalization problems, humans outperformed GPT-3 as well as more recent versions, GPT-3.5 and GPT-4. We don’t know what accounts for the difference from Webb et al.’s results","url":"https://aiguide.substack.com/p/stress-testing-large-language-models","relationship":"Source-stated"},{"text":"Mitchell reports that model accuracy fell substantially on counterfactual tasks using fictional alphabets, while human accuracy stayed relatively high.","excerpt":"the accuracy of humans stays relatively high, while the accuracy of the LLMs drops substantially.","url":"https://aiguide.substack.com/p/stress-testing-large-language-models","relationship":"Source-stated"},{"text":"Webb and coauthors argued that solving permuted-alphabet problems required converting letters to indices; Mitchell rejects that as an account of how humans solve them and as a test of the intended abstract reasoning.","excerpt":"I disagree that the letter-string problems with permuted alphabets “require that letters be converted into the corresponding indices.”","url":"https://aiguide.substack.com/p/stress-testing-large-language-models","relationship":"Source-stated"},{"text":"Mitchell argues that replication and counterfactual tasks matter for evaluating reasoning claims about language models, and calls for scientific debate.","excerpt":"I hope I’ve communicated the importance of doing such experiments, as well as of scientific replication studies. And, of course, of healthy scientific debates!","url":"https://aiguide.substack.com/p/stress-testing-large-language-models","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://aiguide.substack.com/p/stress-testing-large-language-models","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Do the code-assisted results test the same abstract understanding of successorship and predecessorship that Mitchell says the counterfactual tasks were designed to assess?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Melanie Mitchell argues that analogy tests should probe abstract reasoning","summary":"Melanie Mitchell’s post, published May 21, 2024, examines a dispute over whether large language models can reason by analogy. Mitchell and collaborator Martha Lewis report that their replication found humans outperformed GPT-3, GPT-3.5 and GPT-4 on the simplest letter-string problems; Mitchell says the difference from an earlier study remains unexplained. In tests using fictional alphabets and symbols, she reports that human accuracy stayed relatively high while model accuracy dropped substantially, even after the models showed they understood the fictional alphabet’s order.\n\nTaylor Webb and coauthors argued that solving permuted-alphabet problems required converting letters to positions. A GPT-4 version able to generate code used that approach and performed above humans on several types of problems. Mitchell disputes that this tests the same abstract reasoning: she argues people can use successorship and predecessorship without counting positions. That distinction matters because the tasks were designed to probe general analogy, not arithmetic about letters. Mitchell closes by emphasizing stress-testing, replication and scientific debate. The studies leave unresolved what the models’ results say about abstract reasoning, and whether code-assisted solving tests the ability the tasks were intended to assess.","question":"Do the code-assisted results test the same abstract understanding of successorship and predecessorship that Mitchell says the counterfactual tasks were designed to assess?","claims":[{"text":"Mitchell reports that her and Lewis’s replication found humans outperformed GPT-3, GPT-3.5 and GPT-4 on the simplest letter-string analogy problems; she says the difference from the earlier study remains unexplained.","excerpt":"on average, on the zero-generalization problems, humans outperformed GPT-3 as well as more recent versions, GPT-3.5 and GPT-4. We don’t know what accounts for the difference from Webb et al.’s results","support":"source-stated"},{"text":"Mitchell reports that model accuracy fell substantially on counterfactual tasks using fictional alphabets, while human accuracy stayed relatively high.","excerpt":"the accuracy of humans stays relatively high, while the accuracy of the LLMs drops substantially.","support":"source-stated"},{"text":"Webb and coauthors argued that solving permuted-alphabet problems required converting letters to indices; Mitchell rejects that as an account of how humans solve them and as a test of the intended abstract reasoning.","excerpt":"I disagree that the letter-string problems with permuted alphabets “require that letters be converted into the corresponding indices.”","support":"source-stated"},{"text":"Mitchell argues that replication and counterfactual tasks matter for evaluating reasoning claims about language models, and calls for scientific debate.","excerpt":"I hope I’ve communicated the importance of doing such experiments, as well as of scientific replication studies. And, of course, of healthy scientific debates!","support":"source-stated"}],"limitations":["Text as retrieved on October 24, 2026; originally published May 21, 2024.","Mitchell says the differing results between her replication and Webb and coauthors’ study are unexplained, and suggests the participant populations may have contributed. The post does not resolve the disagreement over whether index-based code solving demonstrates the abstract reasoning the tasks were intended to test."],"mediaId":null,"correctionReason":null,"id":"36b0c017c513240ded079a838354981d3de1be29cc225428ec3c1a3b83c2bdf0","publicationId":"4dde8247-ad99-d3b7-4ebc-46833c0861bb","version":1,"sourceUrl":"https://aiguide.substack.com/p/stress-testing-large-language-models","sourceTitle":"Stress-Testing Large Language Models’ Analogical Reasoning Abilities","publisher":"Melanie Mitchell","publishedAt":"2024-05-21T17:42:04.000Z","originalPublished":{"value":"2024-05-21T17:42:04.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-24T01:52:00.948Z","createdAt":"2026-11-18T15:08:09.982Z","historical":true,"entityIds":[],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://aiguide.substack.com/p/stress-testing-large-language-models","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"d7ac5812ee9c1fbf041d5542f670229a15352dd9abd698e864f488cae79ccfc7","rubricVersion":"unrecorded","provider":"unrecorded","model":"anthropic/claude-opus-5.5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-24T01:51:58.529Z"},"provenance":{"textHash":"009a8acbc710627d052d5b1e8460e18ddf3151e3790c3da50da996b7aa32877c","observationId":"473306279a40a6e46d4422a9cf6e739b07cf8b5351a6c7f0a5c43b20a1e46822","retrievedAt":"2026-10-24T02:09:38.316Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"7fbc3393-2b29-74b7-60a6-8dc029eaaf17"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"4cf89ea2-a1b3-dccd-6799-31e52618992d"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-18T15:03:34.564Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/melanie-mitchell-argues-that-analogy-tests-should-probe-36b0c017","formats":{"html":"/readings/melanie-mitchell-argues-that-analogy-tests-should-probe-36b0c017","markdown":"/readings/melanie-mitchell-argues-that-analogy-tests-should-probe-36b0c017.md","json":"/readings/melanie-mitchell-argues-that-analogy-tests-should-probe-36b0c017.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"},{"id":"term-counterfactual","name":"Counterfactual","explanation":"An argument or scenario that considers what would have resulted if events had gone differently from how they actually did, such as if a policy had never been adopted; as an adjective, relating to this kind of thinking about what did not happen but could have.","referenceUrl":"https://dictionary.cambridge.org/dictionary/english/counterfactual"}],"entities":[]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"36b0c017c513240ded079a838354981d3de1be29cc225428ec3c1a3b83c2bdf0","revision":"4dde8247-ad99-d3b7-4ebc-46833c0861bb","revisionKind":"publication-id","title":"Melanie Mitchell argues that analogy tests should probe abstract reasoning","url":"https://escapemeridian.com/readings/melanie-mitchell-argues-that-analogy-tests-should-probe-36b0c017"},"source":{"type":"source","id":"https://aiguide.substack.com/p/stress-testing-large-language-models","revision":"009a8acbc710627d052d5b1e8460e18ddf3151e3790c3da50da996b7aa32877c","revisionKind":"sha256-source-text","title":"Stress-Testing Large Language Models’ Analogical Reasoning Abilities","url":"https://aiguide.substack.com/p/stress-testing-large-language-models"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"36b0c017c513240ded079a838354981d3de1be29cc225428ec3c1a3b83c2bdf0","title":"Melanie Mitchell argues that analogy tests should probe abstract reasoning","argument":"Melanie Mitchell’s post, published May 21, 2024, examines a dispute over whether large language models can reason by analogy. Mitchell and collaborator Martha Lewis report that their replication found humans outperformed GPT-3, GPT-3.5 and GPT-4 on the simplest letter-string problems; Mitchell says the difference from an earlier study remains unexplained. In tests using fictional alphabets and symbols, she reports that human accuracy stayed relatively high while model accuracy dropped substantially, even after the models showed they understood the fictional alphabet’s order.\n\nTaylor Webb and coauthors argued that solving permuted-alphabet problems required converting letters to positions. A GPT-4 version able to generate code used that approach and performed above humans on several types of problems. Mitchell disputes that this tests the same abstract reasoning: she argues people can use successorship and predecessorship without counting positions. That distinction matters because the tasks were designed to probe general analogy, not arithmetic about letters. Mitchell closes by emphasizing stress-testing, replication and scientific debate. The studies leave unresolved what the models’ results say about abstract reasoning, and whether code-assisted solving tests the ability the tasks were intended to assess.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 24, 2026; originally published May 21, 2024. Mitchell says the differing results between her replication and Webb and coauthors’ study are unexplained, and suggests the participant populations may have contributed. The post does not resolve the disagreement over whether index-based code solving demonstrates the abstract reasoning the tasks were intended to test.","speaker":{"name":"Melanie Mitchell","roleAtSource":"Publisher"},"source":{"title":"Stress-Testing Large Language Models’ Analogical Reasoning Abilities","publisher":"Melanie Mitchell","format":"Article","url":"https://aiguide.substack.com/p/stress-testing-large-language-models","publishedAt":"2024-05-21T17:42:04.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"Mitchell reports that her and Lewis’s replication found humans outperformed GPT-3, GPT-3.5 and GPT-4 on the simplest letter-string analogy problems; she says the difference from the earlier study remains unexplained.","excerpt":"on average, on the zero-generalization problems, humans outperformed GPT-3 as well as more recent versions, GPT-3.5 and GPT-4. We don’t know what accounts for the difference from Webb et al.’s results","url":"https://aiguide.substack.com/p/stress-testing-large-language-models","relationship":"Source-stated"},{"text":"Mitchell reports that model accuracy fell substantially on counterfactual tasks using fictional alphabets, while human accuracy stayed relatively high.","excerpt":"the accuracy of humans stays relatively high, while the accuracy of the LLMs drops substantially.","url":"https://aiguide.substack.com/p/stress-testing-large-language-models","relationship":"Source-stated"},{"text":"Webb and coauthors argued that solving permuted-alphabet problems required converting letters to indices; Mitchell rejects that as an account of how humans solve them and as a test of the intended abstract reasoning.","excerpt":"I disagree that the letter-string problems with permuted alphabets “require that letters be converted into the corresponding indices.”","url":"https://aiguide.substack.com/p/stress-testing-large-language-models","relationship":"Source-stated"},{"text":"Mitchell argues that replication and counterfactual tasks matter for evaluating reasoning claims about language models, and calls for scientific debate.","excerpt":"I hope I’ve communicated the importance of doing such experiments, as well as of scientific replication studies. And, of course, of healthy scientific debates!","url":"https://aiguide.substack.com/p/stress-testing-large-language-models","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://aiguide.substack.com/p/stress-testing-large-language-models","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Do the code-assisted results test the same abstract understanding of successorship and predecessorship that Mitchell says the counterfactual tasks were designed to assess?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Melanie Mitchell argues that analogy tests should probe abstract reasoning","summary":"Melanie Mitchell’s post, published May 21, 2024, examines a dispute over whether large language models can reason by analogy. Mitchell and collaborator Martha Lewis report that their replication found humans outperformed GPT-3, GPT-3.5 and GPT-4 on the simplest letter-string problems; Mitchell says the difference from an earlier study remains unexplained. In tests using fictional alphabets and symbols, she reports that human accuracy stayed relatively high while model accuracy dropped substantially, even after the models showed they understood the fictional alphabet’s order.\n\nTaylor Webb and coauthors argued that solving permuted-alphabet problems required converting letters to positions. A GPT-4 version able to generate code used that approach and performed above humans on several types of problems. Mitchell disputes that this tests the same abstract reasoning: she argues people can use successorship and predecessorship without counting positions. That distinction matters because the tasks were designed to probe general analogy, not arithmetic about letters. Mitchell closes by emphasizing stress-testing, replication and scientific debate. The studies leave unresolved what the models’ results say about abstract reasoning, and whether code-assisted solving tests the ability the tasks were intended to assess.","question":"Do the code-assisted results test the same abstract understanding of successorship and predecessorship that Mitchell says the counterfactual tasks were designed to assess?","claims":[{"text":"Mitchell reports that her and Lewis’s replication found humans outperformed GPT-3, GPT-3.5 and GPT-4 on the simplest letter-string analogy problems; she says the difference from the earlier study remains unexplained.","excerpt":"on average, on the zero-generalization problems, humans outperformed GPT-3 as well as more recent versions, GPT-3.5 and GPT-4. We don’t know what accounts for the difference from Webb et al.’s results","support":"source-stated"},{"text":"Mitchell reports that model accuracy fell substantially on counterfactual tasks using fictional alphabets, while human accuracy stayed relatively high.","excerpt":"the accuracy of humans stays relatively high, while the accuracy of the LLMs drops substantially.","support":"source-stated"},{"text":"Webb and coauthors argued that solving permuted-alphabet problems required converting letters to indices; Mitchell rejects that as an account of how humans solve them and as a test of the intended abstract reasoning.","excerpt":"I disagree that the letter-string problems with permuted alphabets “require that letters be converted into the corresponding indices.”","support":"source-stated"},{"text":"Mitchell argues that replication and counterfactual tasks matter for evaluating reasoning claims about language models, and calls for scientific debate.","excerpt":"I hope I’ve communicated the importance of doing such experiments, as well as of scientific replication studies. And, of course, of healthy scientific debates!","support":"source-stated"}],"limitations":["Text as retrieved on October 24, 2026; originally published May 21, 2024.","Mitchell says the differing results between her replication and Webb and coauthors’ study are unexplained, and suggests the participant populations may have contributed. The post does not resolve the disagreement over whether index-based code solving demonstrates the abstract reasoning the tasks were intended to test."],"mediaId":null,"correctionReason":null,"id":"36b0c017c513240ded079a838354981d3de1be29cc225428ec3c1a3b83c2bdf0","publicationId":"4dde8247-ad99-d3b7-4ebc-46833c0861bb","version":1,"sourceUrl":"https://aiguide.substack.com/p/stress-testing-large-language-models","sourceTitle":"Stress-Testing Large Language Models’ Analogical Reasoning Abilities","publisher":"Melanie Mitchell","publishedAt":"2024-05-21T17:42:04.000Z","originalPublished":{"value":"2024-05-21T17:42:04.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-24T01:52:00.948Z","createdAt":"2026-11-18T15:08:09.982Z","historical":true,"entityIds":[],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://aiguide.substack.com/p/stress-testing-large-language-models","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"d7ac5812ee9c1fbf041d5542f670229a15352dd9abd698e864f488cae79ccfc7","rubricVersion":"unrecorded","provider":"unrecorded","model":"anthropic/claude-opus-5.5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-24T01:51:58.529Z"},"provenance":{"textHash":"009a8acbc710627d052d5b1e8460e18ddf3151e3790c3da50da996b7aa32877c","observationId":"473306279a40a6e46d4422a9cf6e739b07cf8b5351a6c7f0a5c43b20a1e46822","retrievedAt":"2026-10-24T02:09:38.316Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"7fbc3393-2b29-74b7-60a6-8dc029eaaf17"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"4cf89ea2-a1b3-dccd-6799-31e52618992d"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-18T15:03:34.564Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"4dde8247-ad99-d3b7-4ebc-46833c0861bb","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"This is a single Substack post by Melanie Mitchell. The reading attributes the findings to Mitchell and Lewis, attributes the counter-argument to Webb and coauthors, and keeps Mitchell's disagreement framed as her own argument.","excerpt":"Webb et al. did not agree with our assessment."},{"dimension":"summary","status":"uncertain","explanation":"Two gaps limit a general reader's understanding. First, 'letter-string problems' is never explained, for example with abc to abcd. Second, the earlier study is never described: Webb et al. found GPT-3 beat UCLA undergraduates, which is the claim Mitchell's replication tests. Taylor Webb also appears later without being identified as that study's author. Smallest repair: one clause with an example problem, and one clause stating that Webb et al. originally reported GPT-3 matching or surpassing humans.","excerpt":"they found that GPT-3 was more accurate (had more correct answers) than the average of a group of UCLA undergraduates the researchers tested."},{"dimension":"date","status":"pass","explanation":"The originalPublished instant 2024-05-21T17:42:04Z matches the dateline May 21, 2024, and the reading states it at day precision. This is a publisher page, not an archive copy. The reading adds no hindsight, and modifiedAt equals the publication time.","excerpt":"May 21, 2024"},{"dimension":"support","status":"uncertain","explanation":"Most claims match the source. One phrase drops Mitchell's hedging: 'even after the models showed they understood the fictional alphabet's order.' Mitchell puts understood in scare quotes, defines it narrowly as naming the next and previous letter, and says the models 'generally' did well. Smallest repair: 'even after checking that the models could generally name each letter's neighbours in the fictional alphabet.'","excerpt":"we made sure that the LLMs “understood” the fictional alphabet, meaning that they could say what the “next letter” and “previous letter” was of each letter in the given fictional alphabet—the LLMs generally did very well on this."},{"dimension":"media","status":"pass","explanation":"No media was selected and no verified candidates were supplied, so there is nothing to check for identity or relevance.","excerpt":null}],"createdAt":"2026-11-18T15:03:34.564Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}