MIT Technology Review · May 20, 2024
MIT Technology Review reports that AI models outperform people on some mental-state tests
Markdown · JSON · Original source
MIT Technology Review’s article was published May 20, 2024. It reports research in Human Behavior comparing GPT-3.5, GPT-4, three versions of Meta’s Llama, and 1,907 people on tests designed to assess theory of mind: inferring other people’s intentions, beliefs, and meanings. GPT-4 scored above human averages on some tasks, while Llama 2’s largest tested model did better than people at recognizing faux pas scenarios; GPT models often answered those scenarios incorrectly. The article stresses that test performance does not mean an AI has human theory of mind. It also reports a concern raised by Maarten Sap: established psychological tests may have appeared in models’ training data, unlike a child encountering a test for the first . That possibility complicates what the scores tell us about the models’ abilities. The article’s conclusion is that such research can help researchers understand what models can and cannot do, while a model’s success on a test does not establish human-like reasoning. How much did possible prior exposure to these tests contribute to the models’ scores?
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
MIT Technology Review · Article
Source details
Original source date: May 20, 2024 (2024-05-20T15:00:00Z).
Date basis: Date from the publisher’s own records, through its API.
Added to Escape Meridian: 2026-11-18T15:18:10.211Z. First observed: 2026-10-28T22:40:01.059Z. Retrieved: 2026-10-28T22:41:34.451Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
Text as retrieved on October 28, 2026; originally published May 20, 2024. The article does not establish how much prior exposure to the tests affected the models’ scores.
Open question
How much did possible prior exposure to these tests contribute to the models’ scores?
Source-stated
MIT Technology Review reports that GPT-4 scored above human averages on several theory-of-mind tasks, while the tested Llama models had mixed results.
Both versions of GPT performed at, or sometimes above, human averages in tasks that involved indirect requests, misdirection, and false beliefs, while GPT-4 outperformed humans in the irony, hinting, and strange stories tests. Llama 2’s three models performed below the human average.
Source-stated
The article reports that Llama 2’s largest tested model outperformed humans at recognizing faux pas scenarios, while GPT consistently gave incorrect responses.
Llama 2, the biggest of the three Meta models tested, outperformed humans when it came to recognizing faux pas scenarios, whereas GPT consistently provided incorrect responses.
Source-stated
The article reports Maarten Sap’s concern that the tests may have appeared in language models’ training data, unlike tests encountered by children.
One reason the LLMs may have performed as well as they did was that these psychological tests are so well established, and were therefore likely to have been included in their training data, says Maarten Sap
Source-stated
The article says that outperforming a human on a theory-of-mind test does not mean an AI has theory of mind.
If an AI outperforms a human on a test designed to measure theory of mind, it does not mean that AI has theory of mind.
Source-stated
The article says this research can help deepen understanding of what language models can and cannot do.
Research like this can help deepen our understanding of what these kinds of models can and cannot do
Source and contributor context
MIT Technology Review · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Nature
Nature publishes research, news, and commentary across the sciences.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 0821168e-aa24-0fa3-dba7-436b2079f464.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T15:13:55.632Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model, second_opinion_categorical.
Observation: 88d79032612856b145446978166ab13cebbb33c3b81a5b6e496c9b97b1f67a1e. Source text hash: 3ecc11033c6a5aca88d8cde29f5ce720c43903d4ce2ac89e8cdcbb1664a740d6.
Revision and source-check history
Version 1 · 2026-11-18T15:18:10.211Z
MIT Technology Review reports that AI models outperform people on some mental-state tests
MIT Technology Review’s article was published May 20, 2024. It reports research in Human Behavior comparing GPT-3.5, GPT-4, three versions of Meta’s Llama, and 1,907 people on tests designed to assess theory of mind: inferring other people’s intentions, beliefs, and meanings. GPT-4 scored above human averages on some tasks, while Llama 2’s largest tested model did better than people at recognizing faux pas scenarios; GPT models often answered those scenarios incorrectly. The article stresses that test performance does not mean an AI has human theory of mind. It also reports a concern raised by Maarten Sap: established psychological tests may have appeared in models’ training data, unlike a child encountering a test for the first . That possibility complicates what the scores tell us about the models’ abilities. The article’s conclusion is that such research can help researchers understand what models can and cannot do, while a model’s success on a test does not establish human-like reasoning. How much did possible prior exposure to these tests contribute to the models’ scores?
Original source
MIT Technology Review · Article
Source details
Original source date: May 20, 2024 (2024-05-20T15:00:00Z).
Date basis: Date from the publisher’s own records, through its API.
Added to Escape Meridian: 2026-11-18T15:18:10.211Z. First observed: 2026-10-28T22:40:01.059Z. Retrieved: 2026-10-28T22:41:34.451Z.
Limits
Text as retrieved on October 28, 2026; originally published May 20, 2024. The article does not establish how much prior exposure to the tests affected the models’ scores.
Open question
How much did possible prior exposure to these tests contribute to the models’ scores?
Source-stated
MIT Technology Review reports that GPT-4 scored above human averages on several theory-of-mind tasks, while the tested Llama models had mixed results.
Both versions of GPT performed at, or sometimes above, human averages in tasks that involved indirect requests, misdirection, and false beliefs, while GPT-4 outperformed humans in the irony, hinting, and strange stories tests. Llama 2’s three models performed below the human average.
Source-stated
The article reports that Llama 2’s largest tested model outperformed humans at recognizing faux pas scenarios, while GPT consistently gave incorrect responses.
Llama 2, the biggest of the three Meta models tested, outperformed humans when it came to recognizing faux pas scenarios, whereas GPT consistently provided incorrect responses.
Source-stated
The article reports Maarten Sap’s concern that the tests may have appeared in language models’ training data, unlike tests encountered by children.
One reason the LLMs may have performed as well as they did was that these psychological tests are so well established, and were therefore likely to have been included in their training data, says Maarten Sap
Source-stated
The article says that outperforming a human on a theory-of-mind test does not mean an AI has theory of mind.
If an AI outperforms a human on a test designed to measure theory of mind, it does not mean that AI has theory of mind.
Source-stated
The article says this research can help deepen understanding of what language models can and cannot do.
Research like this can help deepen our understanding of what these kinds of models can and cannot do
Source and contributor context
MIT Technology Review · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Nature
Nature publishes research, news, and commentary across the sciences.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 0821168e-aa24-0fa3-dba7-436b2079f464.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T15:13:55.632Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model, second_opinion_categorical.
Observation: 88d79032612856b145446978166ab13cebbb33c3b81a5b6e496c9b97b1f67a1e. Source text hash: 3ecc11033c6a5aca88d8cde29f5ce720c43903d4ce2ac89e8cdcbb1664a740d6.
passed · 2026-11-18T15:13:55.632Z · anthropic/claude-opus-5.5
Publication: 0821168e-aa24-0fa3-dba7-436b2079f464
attribution — pass: The reading attributes the report to MIT Technology Review, the research to Nature Human Behavior, and the training-data concern to Maarten Sap, consistent with the text. It avoids the source's ambiguous "he says" quote. The source is a single article, not a listing.
says Maarten Sap, an assistant professor at Carnegie Mellon University, who did not work on the research
summary — pass: The reading explains theory of mind in plain terms, names the models and the human comparison group, and gives the results. It keeps the article's central caveat and Sap's contamination concern. The closing line ties the 'human-like' point to the article's conclusion; that wording echoes Tomer Ullman but matches the article's own framing. The question follows from the evidence.
tasks designed to test the ability to track people’s mental states, known as “theory of mind.”
date — pass: originalPublished is 2024-05-20T15:00Z, and the reading states May 20, 2024, which matches the URL. Later sidebar items, such as the AlphaGo ten-year story, are not used. The page was edited after publication (dateModified 2024-12-05), but every cited passage is plainly part of the main article body. Nothing is framed with hindsight.
New research published today in Nature Human Behavior
support — pass: Each claim matches its excerpt. 'Mixed results' for Llama fairly reflects below-average overall scores plus the largest model's faux pas win. 'Often answered incorrectly' understates the source's 'consistently' but does not contradict it. The line that contamination complicates interpretation is a careful reading of Sap's point. The page was edited after publication, but all cited text is plainly part of the original article.
It’s really important to acknowledge that when you administer a false-belief test to a child, they have probably never seen that exact test before, but language models might
media — pass: No media was selected and no verified candidates were supplied, so no media identity or relevance issue arises.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Nature publishes research, news, and commentary across the sciences.
Explore NatureTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMENature publishes research, news, and commentary across the sciences.
Explore NatureTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIME