{"id":"e348311c0d573b2f2825a58634e74ddfe454dedc5641f685b190a934cef369dd","title":"MIT Technology Review reports that AI models outperform people on some mental-state tests","argument":"MIT Technology Review’s article was published May 20, 2024. It reports research in Nature Human Behavior comparing GPT-3.5, GPT-4, three versions of Meta’s Llama, and 1,907 people on tests designed to assess theory of mind: inferring other people’s intentions, beliefs, and meanings. GPT-4 scored above human averages on some tasks, while Llama 2’s largest tested model did better than people at recognizing faux pas scenarios; GPT models often answered those scenarios incorrectly. The article stresses that test performance does not mean an AI has human theory of mind. It also reports a concern raised by Maarten Sap: established psychological tests may have appeared in models’ training data, unlike a child encountering a test for the first time. That possibility complicates what the scores tell us about the models’ abilities. The article’s conclusion is that such research can help researchers understand what models can and cannot do, while a model’s success on a test does not establish human-like reasoning. How much did possible prior exposure to these tests contribute to the models’ scores?","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 28, 2026; originally published May 20, 2024. The article does not establish how much prior exposure to the tests affected the models’ scores.","speaker":{"name":"MIT Technology Review","roleAtSource":"Publisher"},"source":{"title":"AI models can outperform humans in tests to identify mental states","publisher":"MIT Technology Review","format":"Article","url":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/","publishedAt":"2024-05-20T15:00:00Z","dateBasis":"Date from the publisher’s own records, through its API."},"evidence":[{"text":"MIT Technology Review reports that GPT-4 scored above human averages on several theory-of-mind tasks, while the tested Llama models had mixed results.","excerpt":"Both versions of GPT performed at, or sometimes above, human averages in tasks that involved indirect requests, misdirection, and false beliefs, while GPT-4 outperformed humans in the irony, hinting, and strange stories tests. Llama 2’s three models performed below the human average.","url":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/","relationship":"Source-stated"},{"text":"The article reports that Llama 2’s largest tested model outperformed humans at recognizing faux pas scenarios, while GPT consistently gave incorrect responses.","excerpt":"Llama 2, the biggest of the three Meta models tested, outperformed humans when it came to recognizing faux pas scenarios, whereas GPT consistently provided incorrect responses.","url":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/","relationship":"Source-stated"},{"text":"The article reports Maarten Sap’s concern that the tests may have appeared in language models’ training data, unlike tests encountered by children.","excerpt":"One reason the LLMs may have performed as well as they did was that these psychological tests are so well established, and were therefore likely to have been included in their training data, says Maarten Sap","url":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/","relationship":"Source-stated"},{"text":"The article says that outperforming a human on a theory-of-mind test does not mean an AI has theory of mind.","excerpt":"If an AI outperforms a human on a test designed to measure theory of mind, it does not mean that AI has theory of mind.","url":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/","relationship":"Source-stated"},{"text":"The article says this research can help deepen understanding of what language models can and cannot do.","excerpt":"Research like this can help deepen our understanding of what these kinds of models can and cannot do","url":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How much did possible prior exposure to these tests contribute to the models’ scores?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"MIT Technology Review reports that AI models outperform people on some mental-state tests","summary":"MIT Technology Review’s article was published May 20, 2024. It reports research in Nature Human Behavior comparing GPT-3.5, GPT-4, three versions of Meta’s Llama, and 1,907 people on tests designed to assess theory of mind: inferring other people’s intentions, beliefs, and meanings. GPT-4 scored above human averages on some tasks, while Llama 2’s largest tested model did better than people at recognizing faux pas scenarios; GPT models often answered those scenarios incorrectly. The article stresses that test performance does not mean an AI has human theory of mind. It also reports a concern raised by Maarten Sap: established psychological tests may have appeared in models’ training data, unlike a child encountering a test for the first time. That possibility complicates what the scores tell us about the models’ abilities. The article’s conclusion is that such research can help researchers understand what models can and cannot do, while a model’s success on a test does not establish human-like reasoning. How much did possible prior exposure to these tests contribute to the models’ scores?","question":"How much did possible prior exposure to these tests contribute to the models’ scores?","claims":[{"text":"MIT Technology Review reports that GPT-4 scored above human averages on several theory-of-mind tasks, while the tested Llama models had mixed results.","excerpt":"Both versions of GPT performed at, or sometimes above, human averages in tasks that involved indirect requests, misdirection, and false beliefs, while GPT-4 outperformed humans in the irony, hinting, and strange stories tests. Llama 2’s three models performed below the human average.","support":"source-stated"},{"text":"The article reports that Llama 2’s largest tested model outperformed humans at recognizing faux pas scenarios, while GPT consistently gave incorrect responses.","excerpt":"Llama 2, the biggest of the three Meta models tested, outperformed humans when it came to recognizing faux pas scenarios, whereas GPT consistently provided incorrect responses.","support":"source-stated"},{"text":"The article reports Maarten Sap’s concern that the tests may have appeared in language models’ training data, unlike tests encountered by children.","excerpt":"One reason the LLMs may have performed as well as they did was that these psychological tests are so well established, and were therefore likely to have been included in their training data, says Maarten Sap","support":"source-stated"},{"text":"The article says that outperforming a human on a theory-of-mind test does not mean an AI has theory of mind.","excerpt":"If an AI outperforms a human on a test designed to measure theory of mind, it does not mean that AI has theory of mind.","support":"source-stated"},{"text":"The article says this research can help deepen understanding of what language models can and cannot do.","excerpt":"Research like this can help deepen our understanding of what these kinds of models can and cannot do","support":"source-stated"}],"limitations":["Text as retrieved on October 28, 2026; originally published May 20, 2024.","The article does not establish how much prior exposure to the tests affected the models’ scores."],"mediaId":null,"correctionReason":null,"id":"e348311c0d573b2f2825a58634e74ddfe454dedc5641f685b190a934cef369dd","publicationId":"0821168e-aa24-0fa3-dba7-436b2079f464","version":1,"sourceUrl":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/","sourceTitle":"AI models can outperform humans in tests to identify mental states","publisher":"MIT Technology Review","publishedAt":"2024-05-20T15:00:00Z","originalPublished":{"value":"2024-05-20T15:00:00Z","precision":"instant","basis":"backfill:api_published"},"firstObservedAt":"2026-10-28T22:40:01.059Z","createdAt":"2026-11-18T15:18:10.211Z","historical":true,"entityIds":[],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"4ec9ecc6fa4d754ae036b9b5ca0ec486cefbe19c6e99da2350d0093bd20a8807","rubricVersion":"unrecorded","provider":"anthropic","model":"claude-opus-5-5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-28T22:39:58.306Z"},"provenance":{"textHash":"3ecc11033c6a5aca88d8cde29f5ce720c43903d4ce2ac89e8cdcbb1664a740d6","observationId":"88d79032612856b145446978166ab13cebbb33c3b81a5b6e496c9b97b1f67a1e","retrievedAt":"2026-10-28T22:41:34.451Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"de579340-f795-a9e8-20ca-b4ba34c10b26"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"de9c6cc3-3ce6-c3f0-4fdb-5715941f0237"}},"audit":{"status":"passed","reviewedAt":"2026-11-18T15:13:55.632Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/mit-technology-review-reports-that-ai-models-outperform-people-e348311c","formats":{"html":"/readings/mit-technology-review-reports-that-ai-models-outperform-people-e348311c","markdown":"/readings/mit-technology-review-reports-that-ai-models-outperform-people-e348311c.md","json":"/readings/mit-technology-review-reports-that-ai-models-outperform-people-e348311c.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[],"entities":[{"id":"nature","name":"Nature","kind":"organization","context":"Nature publishes research, news, and commentary across the sciences.","referenceUrls":["https://www.nature.com/","https://www.nature.com/nature.rss"],"path":"/entities/nature","json":"/entities/nature.json","markdown":"/entities/nature.md"},{"id":"time","name":"TIME","kind":"organization","context":"TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.","referenceUrls":["https://time.com/"],"path":"/entities/time","json":"/entities/time.json","markdown":"/entities/time.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"e348311c0d573b2f2825a58634e74ddfe454dedc5641f685b190a934cef369dd","revision":"0821168e-aa24-0fa3-dba7-436b2079f464","revisionKind":"publication-id","title":"MIT Technology Review reports that AI models outperform people on some mental-state tests","url":"https://escapemeridian.com/readings/mit-technology-review-reports-that-ai-models-outperform-people-e348311c"},"source":{"type":"source","id":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/","revision":"3ecc11033c6a5aca88d8cde29f5ce720c43903d4ce2ac89e8cdcbb1664a740d6","revisionKind":"sha256-source-text","title":"AI models can outperform humans in tests to identify mental states","url":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"e348311c0d573b2f2825a58634e74ddfe454dedc5641f685b190a934cef369dd","title":"MIT Technology Review reports that AI models outperform people on some mental-state tests","argument":"MIT Technology Review’s article was published May 20, 2024. It reports research in Nature Human Behavior comparing GPT-3.5, GPT-4, three versions of Meta’s Llama, and 1,907 people on tests designed to assess theory of mind: inferring other people’s intentions, beliefs, and meanings. GPT-4 scored above human averages on some tasks, while Llama 2’s largest tested model did better than people at recognizing faux pas scenarios; GPT models often answered those scenarios incorrectly. The article stresses that test performance does not mean an AI has human theory of mind. It also reports a concern raised by Maarten Sap: established psychological tests may have appeared in models’ training data, unlike a child encountering a test for the first time. That possibility complicates what the scores tell us about the models’ abilities. The article’s conclusion is that such research can help researchers understand what models can and cannot do, while a model’s success on a test does not establish human-like reasoning. How much did possible prior exposure to these tests contribute to the models’ scores?","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 28, 2026; originally published May 20, 2024. The article does not establish how much prior exposure to the tests affected the models’ scores.","speaker":{"name":"MIT Technology Review","roleAtSource":"Publisher"},"source":{"title":"AI models can outperform humans in tests to identify mental states","publisher":"MIT Technology Review","format":"Article","url":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/","publishedAt":"2024-05-20T15:00:00Z","dateBasis":"Date from the publisher’s own records, through its API."},"evidence":[{"text":"MIT Technology Review reports that GPT-4 scored above human averages on several theory-of-mind tasks, while the tested Llama models had mixed results.","excerpt":"Both versions of GPT performed at, or sometimes above, human averages in tasks that involved indirect requests, misdirection, and false beliefs, while GPT-4 outperformed humans in the irony, hinting, and strange stories tests. Llama 2’s three models performed below the human average.","url":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/","relationship":"Source-stated"},{"text":"The article reports that Llama 2’s largest tested model outperformed humans at recognizing faux pas scenarios, while GPT consistently gave incorrect responses.","excerpt":"Llama 2, the biggest of the three Meta models tested, outperformed humans when it came to recognizing faux pas scenarios, whereas GPT consistently provided incorrect responses.","url":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/","relationship":"Source-stated"},{"text":"The article reports Maarten Sap’s concern that the tests may have appeared in language models’ training data, unlike tests encountered by children.","excerpt":"One reason the LLMs may have performed as well as they did was that these psychological tests are so well established, and were therefore likely to have been included in their training data, says Maarten Sap","url":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/","relationship":"Source-stated"},{"text":"The article says that outperforming a human on a theory-of-mind test does not mean an AI has theory of mind.","excerpt":"If an AI outperforms a human on a test designed to measure theory of mind, it does not mean that AI has theory of mind.","url":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/","relationship":"Source-stated"},{"text":"The article says this research can help deepen understanding of what language models can and cannot do.","excerpt":"Research like this can help deepen our understanding of what these kinds of models can and cannot do","url":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How much did possible prior exposure to these tests contribute to the models’ scores?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"MIT Technology Review reports that AI models outperform people on some mental-state tests","summary":"MIT Technology Review’s article was published May 20, 2024. It reports research in Nature Human Behavior comparing GPT-3.5, GPT-4, three versions of Meta’s Llama, and 1,907 people on tests designed to assess theory of mind: inferring other people’s intentions, beliefs, and meanings. GPT-4 scored above human averages on some tasks, while Llama 2’s largest tested model did better than people at recognizing faux pas scenarios; GPT models often answered those scenarios incorrectly. The article stresses that test performance does not mean an AI has human theory of mind. It also reports a concern raised by Maarten Sap: established psychological tests may have appeared in models’ training data, unlike a child encountering a test for the first time. That possibility complicates what the scores tell us about the models’ abilities. The article’s conclusion is that such research can help researchers understand what models can and cannot do, while a model’s success on a test does not establish human-like reasoning. How much did possible prior exposure to these tests contribute to the models’ scores?","question":"How much did possible prior exposure to these tests contribute to the models’ scores?","claims":[{"text":"MIT Technology Review reports that GPT-4 scored above human averages on several theory-of-mind tasks, while the tested Llama models had mixed results.","excerpt":"Both versions of GPT performed at, or sometimes above, human averages in tasks that involved indirect requests, misdirection, and false beliefs, while GPT-4 outperformed humans in the irony, hinting, and strange stories tests. Llama 2’s three models performed below the human average.","support":"source-stated"},{"text":"The article reports that Llama 2’s largest tested model outperformed humans at recognizing faux pas scenarios, while GPT consistently gave incorrect responses.","excerpt":"Llama 2, the biggest of the three Meta models tested, outperformed humans when it came to recognizing faux pas scenarios, whereas GPT consistently provided incorrect responses.","support":"source-stated"},{"text":"The article reports Maarten Sap’s concern that the tests may have appeared in language models’ training data, unlike tests encountered by children.","excerpt":"One reason the LLMs may have performed as well as they did was that these psychological tests are so well established, and were therefore likely to have been included in their training data, says Maarten Sap","support":"source-stated"},{"text":"The article says that outperforming a human on a theory-of-mind test does not mean an AI has theory of mind.","excerpt":"If an AI outperforms a human on a test designed to measure theory of mind, it does not mean that AI has theory of mind.","support":"source-stated"},{"text":"The article says this research can help deepen understanding of what language models can and cannot do.","excerpt":"Research like this can help deepen our understanding of what these kinds of models can and cannot do","support":"source-stated"}],"limitations":["Text as retrieved on October 28, 2026; originally published May 20, 2024.","The article does not establish how much prior exposure to the tests affected the models’ scores."],"mediaId":null,"correctionReason":null,"id":"e348311c0d573b2f2825a58634e74ddfe454dedc5641f685b190a934cef369dd","publicationId":"0821168e-aa24-0fa3-dba7-436b2079f464","version":1,"sourceUrl":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/","sourceTitle":"AI models can outperform humans in tests to identify mental states","publisher":"MIT Technology Review","publishedAt":"2024-05-20T15:00:00Z","originalPublished":{"value":"2024-05-20T15:00:00Z","precision":"instant","basis":"backfill:api_published"},"firstObservedAt":"2026-10-28T22:40:01.059Z","createdAt":"2026-11-18T15:18:10.211Z","historical":true,"entityIds":[],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"4ec9ecc6fa4d754ae036b9b5ca0ec486cefbe19c6e99da2350d0093bd20a8807","rubricVersion":"unrecorded","provider":"anthropic","model":"claude-opus-5-5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-28T22:39:58.306Z"},"provenance":{"textHash":"3ecc11033c6a5aca88d8cde29f5ce720c43903d4ce2ac89e8cdcbb1664a740d6","observationId":"88d79032612856b145446978166ab13cebbb33c3b81a5b6e496c9b97b1f67a1e","retrievedAt":"2026-10-28T22:41:34.451Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"de579340-f795-a9e8-20ca-b4ba34c10b26"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"de9c6cc3-3ce6-c3f0-4fdb-5715941f0237"}},"audit":{"status":"passed","reviewedAt":"2026-11-18T15:13:55.632Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"0821168e-aa24-0fa3-dba7-436b2079f464","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the report to MIT Technology Review, the research to Nature Human Behavior, and the training-data concern to Maarten Sap, consistent with the text. It avoids the source's ambiguous \"he says\" quote. The source is a single article, not a listing.","excerpt":"says Maarten Sap, an assistant professor at Carnegie Mellon University, who did not work on the research"},{"dimension":"summary","status":"pass","explanation":"The reading explains theory of mind in plain terms, names the models and the human comparison group, and gives the results. It keeps the article's central caveat and Sap's contamination concern. The closing line ties the 'human-like' point to the article's conclusion; that wording echoes Tomer Ullman but matches the article's own framing. The question follows from the evidence.","excerpt":"tasks designed to test the ability to track people’s mental states, known as “theory of mind.”"},{"dimension":"date","status":"pass","explanation":"originalPublished is 2024-05-20T15:00Z, and the reading states May 20, 2024, which matches the URL. Later sidebar items, such as the AlphaGo ten-year story, are not used. The page was edited after publication (dateModified 2024-12-05), but every cited passage is plainly part of the main article body. Nothing is framed with hindsight.","excerpt":"New research published today in Nature Human Behavior"},{"dimension":"support","status":"pass","explanation":"Each claim matches its excerpt. 'Mixed results' for Llama fairly reflects below-average overall scores plus the largest model's faux pas win. 'Often answered incorrectly' understates the source's 'consistently' but does not contradict it. The line that contamination complicates interpretation is a careful reading of Sap's point. The page was edited after publication, but all cited text is plainly part of the original article.","excerpt":"It’s really important to acknowledge that when you administer a false-belief test to a child, they have probably never seen that exact test before, but language models might"},{"dimension":"media","status":"pass","explanation":"No media was selected and no verified candidates were supplied, so no media identity or relevance issue arises.","excerpt":null}],"createdAt":"2026-11-18T15:13:55.632Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}