Escape Meridian

Melanie Mitchell · May 2, 2024

Melanie Mitchell argues that benchmark scores do not establish broad human-level abilities

Markdown · JSON · Original source

Melanie Mitchell published this essay on May 2, 2024, challenging a headline that beats humans at basic tasks. Mitchell argues that high scores on —tests designed to compare systems—do not establish that AI surpasses people at the broader abilities those tests are named for. She identifies possible data contamination, reliance on familiar patterns, shortcut learning and weak real-world validity as reasons benchmark results can mislead. For example, she describes studies in which systems used image details or correlated keywords rather than the intended reasoning. Mitchell’s point is not that AI has made little progress: she calls that progress astounding. Rather, she says benchmark performance is not necessarily a good predictor of performance in the real world. She argues that better scientific evaluation should control for shortcuts, test robustness across variations in test items and concepts, and assess how systems perform tasks. The essay’s conclusion is that media reporting should not treat success on a benchmark named for a general ability as success at that ability itself. The evidence here does not include the Nature report, graph or studies Mitchell discusses, so their methods and results cannot be assessed from this text alone.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

Melanie MitchellArticle · aiguide.substack.comOpen original source

Original source

Melanie Mitchell · Article

Source details

Original source date: May 2, 2024 (2024-05-02T21:48:52.000Z).

Date basis: Date supplied by the publisher’s feed.

Added to Escape Meridian: 2026-11-18T19:08:09.855Z. First observed: 2026-10-23T20:21:42.773Z. Retrieved: 2026-10-23T20:23:37.322Z.

Referenced by

Recorded citations in public readings. A reference alone does not establish agreement or influence.

No public references recorded here yet.

Reference data (JSON)

Limits

Text as retrieved on October 23, 2026; originally published May 2, 2024. The report, graph and studies Mitchell discusses are not included in the supplied text, so their methods and results cannot be assessed here.

Open question

How would evaluation methods that control for shortcuts and test robustness across variations change the assessment of AI systems’ performance on these abilities?

Source-stated

Nature’s headline said AI now beats humans at basic tasks, according to Mitchell.

Two weeks ago, Nature, one of the world’s most prestigious journals, had this jarring headline:

Source-stated

Mitchell argues that success on a benchmark named for a general ability does not establish success at that ability.

AI surpassing humans on a benchmark that is named after a general ability is not the same as AI surpassing humans on that general ability.

Source-stated

Mitchell says benchmark scores can mislead for reasons including data contamination, reliance on familiar patterns, shortcut learning and test validity.

There are at least four reasons that AI performance on a benchmark can be misleading.

Source-stated

Mitchell calls for better scientific evaluation that controls for shortcuts, tests robustness across variations, and assesses how machines perform tasks.

ones that control for shortcuts, that test robustness over variations on both the form of test items and on the underlying concepts being assessed, along with other ways to assess the mechanisms by which machines are performing tasks.

Source and contributor context

Melanie Mitchell · Publisher

The publisher is identified here. No individual author has been inferred.

Source media

Media retrieval status: unchecked.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

Nature

Nature publishes research, news, and commentary across the sciences.

AI Now Institute

AI Now Institute studies the power and institutions shaping AI and advocates for public accountability.

Editorial provenance

Version 1. Publication ID: d31de991-524b-3028-4924-5790ab3f61de.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T19:04:14.562Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge_admitted_by_model.

Observation: d57081a018d93a3d60a867805c3895ea56cc6e912cbf583468dbfe1a0ee4f9e9. Source text hash: 20e8a5ad0979fbd57a70f25b90961fcf60871f22327de8b3180b7de5132bd4c3.

Revision and source-check history

Version 1 · 2026-11-18T19:08:09.855Z

Melanie Mitchell argues that benchmark scores do not establish broad human-level abilities

Melanie Mitchell published this essay on May 2, 2024, challenging a headline that beats humans at basic tasks. Mitchell argues that high scores on —tests designed to compare systems—do not establish that AI surpasses people at the broader abilities those tests are named for. She identifies possible data contamination, reliance on familiar patterns, shortcut learning and weak real-world validity as reasons benchmark results can mislead. For example, she describes studies in which systems used image details or correlated keywords rather than the intended reasoning. Mitchell’s point is not that AI has made little progress: she calls that progress astounding. Rather, she says benchmark performance is not necessarily a good predictor of performance in the real world. She argues that better scientific evaluation should control for shortcuts, test robustness across variations in test items and concepts, and assess how systems perform tasks. The essay’s conclusion is that media reporting should not treat success on a benchmark named for a general ability as success at that ability itself. The evidence here does not include the Nature report, graph or studies Mitchell discusses, so their methods and results cannot be assessed from this text alone.

Original source

Melanie Mitchell · Article

Source details

Original source date: May 2, 2024 (2024-05-02T21:48:52.000Z).

Date basis: Date supplied by the publisher’s feed.

Added to Escape Meridian: 2026-11-18T19:08:09.855Z. First observed: 2026-10-23T20:21:42.773Z. Retrieved: 2026-10-23T20:23:37.322Z.

Limits

Text as retrieved on October 23, 2026; originally published May 2, 2024. The report, graph and studies Mitchell discusses are not included in the supplied text, so their methods and results cannot be assessed here.

Open question

How would evaluation methods that control for shortcuts and test robustness across variations change the assessment of AI systems’ performance on these abilities?

Source-stated

Nature’s headline said AI now beats humans at basic tasks, according to Mitchell.

Two weeks ago, Nature, one of the world’s most prestigious journals, had this jarring headline:

Source-stated

Mitchell argues that success on a benchmark named for a general ability does not establish success at that ability.

AI surpassing humans on a benchmark that is named after a general ability is not the same as AI surpassing humans on that general ability.

Source-stated

Mitchell says benchmark scores can mislead for reasons including data contamination, reliance on familiar patterns, shortcut learning and test validity.

There are at least four reasons that AI performance on a benchmark can be misleading.

Source-stated

Mitchell calls for better scientific evaluation that controls for shortcuts, tests robustness across variations, and assesses how machines perform tasks.

ones that control for shortcuts, that test robustness over variations on both the form of test items and on the underlying concepts being assessed, along with other ways to assess the mechanisms by which machines are performing tasks.

Source and contributor context

Melanie Mitchell · Publisher

The publisher is identified here. No individual author has been inferred.

Source media

Media retrieval status: unchecked.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

Nature

Nature publishes research, news, and commentary across the sciences.

AI Now Institute

AI Now Institute studies the power and institutions shaping AI and advocates for public accountability.

Editorial provenance

Version 1. Publication ID: d31de991-524b-3028-4924-5790ab3f61de.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T19:04:14.562Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge_admitted_by_model.

Observation: d57081a018d93a3d60a867805c3895ea56cc6e912cbf583468dbfe1a0ee4f9e9. Source text hash: 20e8a5ad0979fbd57a70f25b90961fcf60871f22327de8b3180b7de5132bd4c3.

passed · 2026-11-18T19:04:14.562Z · anthropic/claude-opus-5.5

Publication: d31de991-524b-3028-4924-5790ab3f61de

attribution — pass: The page is a single essay by Melanie Mitchell, and the reading attributes its arguments to her. Nature's headline is attributed to Nature as Mitchell reports it, and the claims about studies are framed as her descriptions.

But let me repeat an important mantra:

summary — pass: The reading leads with Mitchell's challenge to the Nature headline. It explains benchmarks in plain words, and a definition is also supplied. It covers her four reasons, the examples, her acknowledgement of astounding progress, her proposed evaluation methods and her mantra for the media. Data contamination is named without a gloss, but it is one item in a list and does not block understanding. The voice and force of the essay are preserved. The question follows from her call for better evaluation.

AI has made astounding progress, but the assumption that the Nature headline makes is not correct.

date — pass: The reading's date of May 2, 2024 matches the source dateline and the UTC calendar day of originalPublished. The evidence kind is publisher-page, not archive-copy, so no archive disclosure is required. The modified date equals the publication date. There is no hindsight framing and no use of later material.

Melanie Mitchell May 02, 2024

support — pass: Each claim is source-stated and grounded in the text. These cover the headline, the mantra, the four reasons, the evaluation proposals and the point that benchmark performance is not necessarily a good real-world predictor. The examples of image cues and correlated keywords match the ruler and natural-language-inference studies. The limitation about the absent Nature report and studies is accurate. No benefits or costs are invented.

Again and again, it’s been shown that AI performance on benchmarks is not necessarily a good predictor for AI performance in the real world.

media — pass: No media was selected, and no verified media candidates were supplied. This is appropriate.

Challenge or correct this reading

Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.