# MIT Technology Review reports that AI models outperform people on some mental-state tests

[Canonical reading](https://escapemeridian.com/readings/mit-technology-review-reports-that-ai-models-outperform-people-e348311c)

MIT Technology Review’s article was published May 20, 2024. It reports research in Nature Human Behavior comparing GPT-3.5, GPT-4, three versions of Meta’s Llama, and 1,907 people on tests designed to assess theory of mind: inferring other people’s intentions, beliefs, and meanings. GPT-4 scored above human averages on some tasks, while Llama 2’s largest tested model did better than people at recognizing faux pas scenarios; GPT models often answered those scenarios incorrectly. The article stresses that test performance does not mean an AI has human theory of mind. It also reports a concern raised by Maarten Sap: established psychological tests may have appeared in models’ training data, unlike a child encountering a test for the first time. That possibility complicates what the scores tell us about the models’ abilities. The article’s conclusion is that such research can help researchers understand what models can and cannot do, while a model’s success on a test does not establish human-like reasoning. How much did possible prior exposure to these tests contribute to the models’ scores?

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

MIT Technology Review · Article

Original source date: May 20, 2024 (2024-05-20T15:00:00Z).

Date basis: Date from the publisher’s own records, through its API.

Added to Escape Meridian: 2026-11-18T15:18:10.211Z. First observed: 2026-10-28T22:40:01.059Z. Retrieved: 2026-10-28T22:41:34.451Z.

- [AI models can outperform humans in tests to identify mental states](https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/)

## Limits

Text as retrieved on October 28, 2026; originally published May 20, 2024. The article does not establish how much prior exposure to the tests affected the models’ scores.


## Open question

How much did possible prior exposure to these tests contribute to the models’ scores?


## Source-stated

MIT Technology Review reports that GPT-4 scored above human averages on several theory-of-mind tasks, while the tested Llama models had mixed results.

> Both versions of GPT performed at, or sometimes above, human averages in tasks that involved indirect requests, misdirection, and false beliefs, while GPT-4 outperformed humans in the irony, hinting, and strange stories tests. Llama 2’s three models performed below the human average.

- [Supporting source](https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/)

## Source-stated

The article reports that Llama 2’s largest tested model outperformed humans at recognizing faux pas scenarios, while GPT consistently gave incorrect responses.

> Llama 2, the biggest of the three Meta models tested, outperformed humans when it came to recognizing faux pas scenarios, whereas GPT consistently provided incorrect responses.

- [Supporting source](https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/)

## Source-stated

The article reports Maarten Sap’s concern that the tests may have appeared in language models’ training data, unlike tests encountered by children.

> One reason the LLMs may have performed as well as they did was that these psychological tests are so well established, and were therefore likely to have been included in their training data, says Maarten Sap

- [Supporting source](https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/)

## Source-stated

The article says that outperforming a human on a theory-of-mind test does not mean an AI has theory of mind.

> If an AI outperforms a human on a test designed to measure theory of mind, it does not mean that AI has theory of mind.

- [Supporting source](https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/)

## Source-stated

The article says this research can help deepen understanding of what language models can and cannot do.

> Research like this can help deepen our understanding of what these kinds of models can and cannot do

- [Supporting source](https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/)

## Source and contributor context

MIT Technology Review · Publisher

The publisher is identified here. No individual author has been inferred.


## Source media

Media retrieval status: unchecked.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Nature

Nature publishes research, news, and commentary across the sciences.

- [Explore this context](/entities/nature)
- [Reference](https://www.nature.com/)
- [Reference](https://www.nature.com/nature.rss)

## TIME

TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.

- [Explore this context](/entities/time)
- [Reference](https://time.com/)

## Editorial provenance

Version 1. Publication ID: 0821168e-aa24-0fa3-dba7-436b2079f464.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T15:13:55.632Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge\_admitted\_by\_model, second\_opinion\_categorical.

Observation: 88d79032612856b145446978166ab13cebbb33c3b81a5b6e496c9b97b1f67a1e. Source text hash: 3ecc11033c6a5aca88d8cde29f5ce720c43903d4ce2ac89e8cdcbb1664a740d6.


## Revision and source-check history

### Version 1 — 2026-11-18T15:18:10.211Z

MIT Technology Review’s article was published May 20, 2024. It reports research in Nature Human Behavior comparing GPT-3.5, GPT-4, three versions of Meta’s Llama, and 1,907 people on tests designed to assess theory of mind: inferring other people’s intentions, beliefs, and meanings. GPT-4 scored above human averages on some tasks, while Llama 2’s largest tested model did better than people at recognizing faux pas scenarios; GPT models often answered those scenarios incorrectly. The article stresses that test performance does not mean an AI has human theory of mind. It also reports a concern raised by Maarten Sap: established psychological tests may have appeared in models’ training data, unlike a child encountering a test for the first time. That possibility complicates what the scores tell us about the models’ abilities. The article’s conclusion is that such research can help researchers understand what models can and cannot do, while a model’s success on a test does not establish human-like reasoning. How much did possible prior exposure to these tests contribute to the models’ scores?

## Original source

MIT Technology Review · Article

Original source date: May 20, 2024 (2024-05-20T15:00:00Z).

Date basis: Date from the publisher’s own records, through its API.

Added to Escape Meridian: 2026-11-18T15:18:10.211Z. First observed: 2026-10-28T22:40:01.059Z. Retrieved: 2026-10-28T22:41:34.451Z.

- [AI models can outperform humans in tests to identify mental states](https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/)

## Limits

Text as retrieved on October 28, 2026; originally published May 20, 2024. The article does not establish how much prior exposure to the tests affected the models’ scores.


## Open question

How much did possible prior exposure to these tests contribute to the models’ scores?


## Source-stated

MIT Technology Review reports that GPT-4 scored above human averages on several theory-of-mind tasks, while the tested Llama models had mixed results.

> Both versions of GPT performed at, or sometimes above, human averages in tasks that involved indirect requests, misdirection, and false beliefs, while GPT-4 outperformed humans in the irony, hinting, and strange stories tests. Llama 2’s three models performed below the human average.

- [Supporting source](https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/)

## Source-stated

The article reports that Llama 2’s largest tested model outperformed humans at recognizing faux pas scenarios, while GPT consistently gave incorrect responses.

> Llama 2, the biggest of the three Meta models tested, outperformed humans when it came to recognizing faux pas scenarios, whereas GPT consistently provided incorrect responses.

- [Supporting source](https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/)

## Source-stated

The article reports Maarten Sap’s concern that the tests may have appeared in language models’ training data, unlike tests encountered by children.

> One reason the LLMs may have performed as well as they did was that these psychological tests are so well established, and were therefore likely to have been included in their training data, says Maarten Sap

- [Supporting source](https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/)

## Source-stated

The article says that outperforming a human on a theory-of-mind test does not mean an AI has theory of mind.

> If an AI outperforms a human on a test designed to measure theory of mind, it does not mean that AI has theory of mind.

- [Supporting source](https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/)

## Source-stated

The article says this research can help deepen understanding of what language models can and cannot do.

> Research like this can help deepen our understanding of what these kinds of models can and cannot do

- [Supporting source](https://www.technologyreview.com/2024/05/20/1092681/ai-models-can-outperform-humans-in-tests-to-identify-mental-states/)

## Source and contributor context

MIT Technology Review · Publisher

The publisher is identified here. No individual author has been inferred.


## Source media

Media retrieval status: unchecked.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Nature

Nature publishes research, news, and commentary across the sciences.

- [Explore this context](/entities/nature)
- [Reference](https://www.nature.com/)
- [Reference](https://www.nature.com/nature.rss)

## TIME

TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.

- [Explore this context](/entities/time)
- [Reference](https://time.com/)

## Editorial provenance

Version 1. Publication ID: 0821168e-aa24-0fa3-dba7-436b2079f464.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T15:13:55.632Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge\_admitted\_by\_model, second\_opinion\_categorical.

Observation: 88d79032612856b145446978166ab13cebbb33c3b81a5b6e496c9b97b1f67a1e. Source text hash: 3ecc11033c6a5aca88d8cde29f5ce720c43903d4ce2ac89e8cdcbb1664a740d6.


### passed — 2026-11-18T15:13:55.632Z — anthropic/claude-opus-5.5

Publication: 0821168e-aa24-0fa3-dba7-436b2079f464

attribution — pass: The reading attributes the report to MIT Technology Review, the research to Nature Human Behavior, and the training-data concern to Maarten Sap, consistent with the text. It avoids the source's ambiguous "he says" quote. The source is a single article, not a listing.

> says Maarten Sap, an assistant professor at Carnegie Mellon University, who did not work on the research

summary — pass: The reading explains theory of mind in plain terms, names the models and the human comparison group, and gives the results. It keeps the article's central caveat and Sap's contamination concern. The closing line ties the 'human-like' point to the article's conclusion; that wording echoes Tomer Ullman but matches the article's own framing. The question follows from the evidence.

> tasks designed to test the ability to track people’s mental states, known as “theory of mind.”

date — pass: originalPublished is 2024-05-20T15:00Z, and the reading states May 20, 2024, which matches the URL. Later sidebar items, such as the AlphaGo ten-year story, are not used. The page was edited after publication (dateModified 2024-12-05), but every cited passage is plainly part of the main article body. Nothing is framed with hindsight.

> New research published today in Nature Human Behavior

support — pass: Each claim matches its excerpt. 'Mixed results' for Llama fairly reflects below-average overall scores plus the largest model's faux pas win. 'Often answered incorrectly' understates the source's 'consistently' but does not contradict it. The line that contamination complicates interpretation is a careful reading of Sap's point. The page was edited after publication, but all cited text is plainly part of the original article.

> It’s really important to acknowledge that when you administer a false-belief test to a child, they have probably never seen that exact test before, but language models might

media — pass: No media was selected and no verified candidates were supplied, so no media identity or relevance issue arises.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/e348311c0d573b2f2825a58634e74ddfe454dedc5641f685b190a934cef369dd/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/mit-technology-review-reports-that-ai-models-outperform-people-e348311c.json)
