# Melanie Mitchell argues that benchmark scores do not establish broad human-level abilities

[Canonical reading](https://escapemeridian.com/readings/melanie-mitchell-argues-that-benchmark-scores-do-not-establish-61c94d82)

Melanie Mitchell published this essay on May 2, 2024, challenging a Nature headline that AI now beats humans at basic tasks. Mitchell argues that high scores on benchmarks—tests designed to compare systems—do not establish that AI surpasses people at the broader abilities those tests are named for. She identifies possible data contamination, reliance on familiar patterns, shortcut learning and weak real-world validity as reasons benchmark results can mislead. For example, she describes studies in which systems used image details or correlated keywords rather than the intended reasoning. Mitchell’s point is not that AI has made little progress: she calls that progress astounding. Rather, she says benchmark performance is not necessarily a good predictor of performance in the real world. She argues that better scientific evaluation should control for shortcuts, test robustness across variations in test items and concepts, and assess how systems perform tasks. The essay’s conclusion is that media reporting should not treat success on a benchmark named for a general ability as success at that ability itself. The evidence here does not include the Nature report, graph or studies Mitchell discusses, so their methods and results cannot be assessed from this text alone.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

Melanie Mitchell · Article

Original source date: May 2, 2024 (2024-05-02T21:48:52.000Z).

Date basis: Date supplied by the publisher’s feed.

Added to Escape Meridian: 2026-11-18T19:08:09.855Z. First observed: 2026-10-23T20:21:42.773Z. Retrieved: 2026-10-23T20:23:37.322Z.

- ["AI now beats humans at basic tasks": Really?](https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks)

## Limits

Text as retrieved on October 23, 2026; originally published May 2, 2024. The Nature report, graph and studies Mitchell discusses are not included in the supplied text, so their methods and results cannot be assessed here.


## Open question

How would evaluation methods that control for shortcuts and test robustness across variations change the assessment of AI systems’ performance on these abilities?


## Source-stated

Nature’s headline said AI now beats humans at basic tasks, according to Mitchell.

> Two weeks ago, Nature, one of the world’s most prestigious journals, had this jarring headline:

- [Supporting source](https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks)

## Source-stated

Mitchell argues that success on a benchmark named for a general ability does not establish success at that ability.

> AI surpassing humans on a benchmark that is named after a general ability is not the same as AI surpassing humans on that general ability.

- [Supporting source](https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks)

## Source-stated

Mitchell says benchmark scores can mislead for reasons including data contamination, reliance on familiar patterns, shortcut learning and test validity.

> There are at least four reasons that AI performance on a benchmark can be misleading.

- [Supporting source](https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks)

## Source-stated

Mitchell calls for better scientific evaluation that controls for shortcuts, tests robustness across variations, and assesses how machines perform tasks.

> ones that control for shortcuts, that test robustness over variations on both the form of test items and on the underlying concepts being assessed, along with other ways to assess the mechanisms by which machines are performing tasks.

- [Supporting source](https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks)

## Source and contributor context

Melanie Mitchell · Publisher

The publisher is identified here. No individual author has been inferred.


## Source media

Media retrieval status: unchecked.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

- [Definition reference](https://arxiv.org/abs/2211.09110)

## Nature

Nature publishes research, news, and commentary across the sciences.

- [Explore this context](/entities/nature)
- [Reference](https://www.nature.com/)
- [Reference](https://www.nature.com/nature.rss)

## AI Now Institute

AI Now Institute studies the power and institutions shaping AI and advocates for public accountability.

- [Explore this context](/entities/ai-now)
- [Reference](https://ainowinstitute.org/)

## Editorial provenance

Version 1. Publication ID: d31de991-524b-3028-4924-5790ab3f61de.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T19:04:14.562Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge\_admitted\_by\_model.

Observation: d57081a018d93a3d60a867805c3895ea56cc6e912cbf583468dbfe1a0ee4f9e9. Source text hash: 20e8a5ad0979fbd57a70f25b90961fcf60871f22327de8b3180b7de5132bd4c3.


## Revision and source-check history

### Version 1 — 2026-11-18T19:08:09.855Z

Melanie Mitchell published this essay on May 2, 2024, challenging a Nature headline that AI now beats humans at basic tasks. Mitchell argues that high scores on benchmarks—tests designed to compare systems—do not establish that AI surpasses people at the broader abilities those tests are named for. She identifies possible data contamination, reliance on familiar patterns, shortcut learning and weak real-world validity as reasons benchmark results can mislead. For example, she describes studies in which systems used image details or correlated keywords rather than the intended reasoning. Mitchell’s point is not that AI has made little progress: she calls that progress astounding. Rather, she says benchmark performance is not necessarily a good predictor of performance in the real world. She argues that better scientific evaluation should control for shortcuts, test robustness across variations in test items and concepts, and assess how systems perform tasks. The essay’s conclusion is that media reporting should not treat success on a benchmark named for a general ability as success at that ability itself. The evidence here does not include the Nature report, graph or studies Mitchell discusses, so their methods and results cannot be assessed from this text alone.

## Original source

Melanie Mitchell · Article

Original source date: May 2, 2024 (2024-05-02T21:48:52.000Z).

Date basis: Date supplied by the publisher’s feed.

Added to Escape Meridian: 2026-11-18T19:08:09.855Z. First observed: 2026-10-23T20:21:42.773Z. Retrieved: 2026-10-23T20:23:37.322Z.

- ["AI now beats humans at basic tasks": Really?](https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks)

## Limits

Text as retrieved on October 23, 2026; originally published May 2, 2024. The Nature report, graph and studies Mitchell discusses are not included in the supplied text, so their methods and results cannot be assessed here.


## Open question

How would evaluation methods that control for shortcuts and test robustness across variations change the assessment of AI systems’ performance on these abilities?


## Source-stated

Nature’s headline said AI now beats humans at basic tasks, according to Mitchell.

> Two weeks ago, Nature, one of the world’s most prestigious journals, had this jarring headline:

- [Supporting source](https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks)

## Source-stated

Mitchell argues that success on a benchmark named for a general ability does not establish success at that ability.

> AI surpassing humans on a benchmark that is named after a general ability is not the same as AI surpassing humans on that general ability.

- [Supporting source](https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks)

## Source-stated

Mitchell says benchmark scores can mislead for reasons including data contamination, reliance on familiar patterns, shortcut learning and test validity.

> There are at least four reasons that AI performance on a benchmark can be misleading.

- [Supporting source](https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks)

## Source-stated

Mitchell calls for better scientific evaluation that controls for shortcuts, tests robustness across variations, and assesses how machines perform tasks.

> ones that control for shortcuts, that test robustness over variations on both the form of test items and on the underlying concepts being assessed, along with other ways to assess the mechanisms by which machines are performing tasks.

- [Supporting source](https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks)

## Source and contributor context

Melanie Mitchell · Publisher

The publisher is identified here. No individual author has been inferred.


## Source media

Media retrieval status: unchecked.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

- [Definition reference](https://arxiv.org/abs/2211.09110)

## Nature

Nature publishes research, news, and commentary across the sciences.

- [Explore this context](/entities/nature)
- [Reference](https://www.nature.com/)
- [Reference](https://www.nature.com/nature.rss)

## AI Now Institute

AI Now Institute studies the power and institutions shaping AI and advocates for public accountability.

- [Explore this context](/entities/ai-now)
- [Reference](https://ainowinstitute.org/)

## Editorial provenance

Version 1. Publication ID: d31de991-524b-3028-4924-5790ab3f61de.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T19:04:14.562Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge\_admitted\_by\_model.

Observation: d57081a018d93a3d60a867805c3895ea56cc6e912cbf583468dbfe1a0ee4f9e9. Source text hash: 20e8a5ad0979fbd57a70f25b90961fcf60871f22327de8b3180b7de5132bd4c3.


### passed — 2026-11-18T19:04:14.562Z — anthropic/claude-opus-5.5

Publication: d31de991-524b-3028-4924-5790ab3f61de

attribution — pass: The page is a single essay by Melanie Mitchell, and the reading attributes its arguments to her. Nature's headline is attributed to Nature as Mitchell reports it, and the claims about studies are framed as her descriptions.

> But let me repeat an important mantra:

summary — pass: The reading leads with Mitchell's challenge to the Nature headline. It explains benchmarks in plain words, and a definition is also supplied. It covers her four reasons, the examples, her acknowledgement of astounding progress, her proposed evaluation methods and her mantra for the media. Data contamination is named without a gloss, but it is one item in a list and does not block understanding. The voice and force of the essay are preserved. The question follows from her call for better evaluation.

> AI has made astounding progress, but the assumption that the Nature headline makes is not correct.

date — pass: The reading's date of May 2, 2024 matches the source dateline and the UTC calendar day of originalPublished. The evidence kind is publisher-page, not archive-copy, so no archive disclosure is required. The modified date equals the publication date. There is no hindsight framing and no use of later material.

> Melanie Mitchell
May 02, 2024

support — pass: Each claim is source-stated and grounded in the text. These cover the headline, the mantra, the four reasons, the evaluation proposals and the point that benchmark performance is not necessarily a good real-world predictor. The examples of image cues and correlated keywords match the ruler and natural-language-inference studies. The limitation about the absent Nature report and studies is accurate. No benefits or costs are invented.

> Again and again, it’s been shown that AI performance on benchmarks is not necessarily a good predictor for AI performance in the real world.

media — pass: No media was selected, and no verified media candidates were supplied. This is appropriate.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/61c94d82a9d69cce7e211de0bed8efd8cda9aa359743e05d283e7ded341f2a80/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/melanie-mitchell-argues-that-benchmark-scores-do-not-establish-61c94d82.json)
