{"id":"61c94d82a9d69cce7e211de0bed8efd8cda9aa359743e05d283e7ded341f2a80","title":"Melanie Mitchell argues that benchmark scores do not establish broad human-level abilities","argument":"Melanie Mitchell published this essay on May 2, 2024, challenging a Nature headline that AI now beats humans at basic tasks. Mitchell argues that high scores on benchmarks—tests designed to compare systems—do not establish that AI surpasses people at the broader abilities those tests are named for. She identifies possible data contamination, reliance on familiar patterns, shortcut learning and weak real-world validity as reasons benchmark results can mislead. For example, she describes studies in which systems used image details or correlated keywords rather than the intended reasoning. Mitchell’s point is not that AI has made little progress: she calls that progress astounding. Rather, she says benchmark performance is not necessarily a good predictor of performance in the real world. She argues that better scientific evaluation should control for shortcuts, test robustness across variations in test items and concepts, and assess how systems perform tasks. The essay’s conclusion is that media reporting should not treat success on a benchmark named for a general ability as success at that ability itself. The evidence here does not include the Nature report, graph or studies Mitchell discusses, so their methods and results cannot be assessed from this text alone.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 23, 2026; originally published May 2, 2024. The Nature report, graph and studies Mitchell discusses are not included in the supplied text, so their methods and results cannot be assessed here.","speaker":{"name":"Melanie Mitchell","roleAtSource":"Publisher"},"source":{"title":"\"AI now beats humans at basic tasks\": Really?","publisher":"Melanie Mitchell","format":"Article","url":"https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks","publishedAt":"2024-05-02T21:48:52.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"Nature’s headline said AI now beats humans at basic tasks, according to Mitchell.","excerpt":"Two weeks ago, Nature, one of the world’s most prestigious journals, had this jarring headline:","url":"https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks","relationship":"Source-stated"},{"text":"Mitchell argues that success on a benchmark named for a general ability does not establish success at that ability.","excerpt":"AI surpassing humans on a benchmark that is named after a general ability is not the same as AI surpassing humans on that general ability.","url":"https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks","relationship":"Source-stated"},{"text":"Mitchell says benchmark scores can mislead for reasons including data contamination, reliance on familiar patterns, shortcut learning and test validity.","excerpt":"There are at least four reasons that AI performance on a benchmark can be misleading.","url":"https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks","relationship":"Source-stated"},{"text":"Mitchell calls for better scientific evaluation that controls for shortcuts, tests robustness across variations, and assesses how machines perform tasks.","excerpt":"ones that control for shortcuts, that test robustness over variations on both the form of test items and on the underlying concepts being assessed, along with other ways to assess the mechanisms by which machines are performing tasks.","url":"https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How would evaluation methods that control for shortcuts and test robustness across variations change the assessment of AI systems’ performance on these abilities?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Melanie Mitchell argues that benchmark scores do not establish broad human-level abilities","summary":"Melanie Mitchell published this essay on May 2, 2024, challenging a Nature headline that AI now beats humans at basic tasks. Mitchell argues that high scores on benchmarks—tests designed to compare systems—do not establish that AI surpasses people at the broader abilities those tests are named for. She identifies possible data contamination, reliance on familiar patterns, shortcut learning and weak real-world validity as reasons benchmark results can mislead. For example, she describes studies in which systems used image details or correlated keywords rather than the intended reasoning. Mitchell’s point is not that AI has made little progress: she calls that progress astounding. Rather, she says benchmark performance is not necessarily a good predictor of performance in the real world. She argues that better scientific evaluation should control for shortcuts, test robustness across variations in test items and concepts, and assess how systems perform tasks. The essay’s conclusion is that media reporting should not treat success on a benchmark named for a general ability as success at that ability itself. The evidence here does not include the Nature report, graph or studies Mitchell discusses, so their methods and results cannot be assessed from this text alone.","question":"How would evaluation methods that control for shortcuts and test robustness across variations change the assessment of AI systems’ performance on these abilities?","claims":[{"text":"Nature’s headline said AI now beats humans at basic tasks, according to Mitchell.","excerpt":"Two weeks ago, Nature, one of the world’s most prestigious journals, had this jarring headline:","support":"source-stated"},{"text":"Mitchell argues that success on a benchmark named for a general ability does not establish success at that ability.","excerpt":"AI surpassing humans on a benchmark that is named after a general ability is not the same as AI surpassing humans on that general ability.","support":"source-stated"},{"text":"Mitchell says benchmark scores can mislead for reasons including data contamination, reliance on familiar patterns, shortcut learning and test validity.","excerpt":"There are at least four reasons that AI performance on a benchmark can be misleading.","support":"source-stated"},{"text":"Mitchell calls for better scientific evaluation that controls for shortcuts, tests robustness across variations, and assesses how machines perform tasks.","excerpt":"ones that control for shortcuts, that test robustness over variations on both the form of test items and on the underlying concepts being assessed, along with other ways to assess the mechanisms by which machines are performing tasks.","support":"source-stated"}],"limitations":["Text as retrieved on October 23, 2026; originally published May 2, 2024.","The Nature report, graph and studies Mitchell discusses are not included in the supplied text, so their methods and results cannot be assessed here."],"mediaId":null,"correctionReason":null,"id":"61c94d82a9d69cce7e211de0bed8efd8cda9aa359743e05d283e7ded341f2a80","publicationId":"d31de991-524b-3028-4924-5790ab3f61de","version":1,"sourceUrl":"https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks","sourceTitle":"\"AI now beats humans at basic tasks\": Really?","publisher":"Melanie Mitchell","publishedAt":"2024-05-02T21:48:52.000Z","originalPublished":{"value":"2024-05-02T21:48:52.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-23T20:21:42.773Z","createdAt":"2026-11-18T19:08:09.855Z","historical":true,"entityIds":[],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"63bd5df7730c5a253850dd0633fa5dc409686e050c23c820771dc51a4e7a429f","rubricVersion":"em-backfill-context/1","provider":"vercel","model":"typesafe-ai/jev","reasonCodes":["bridge_admitted_by_model"],"confidence":0.55,"decidedAt":"2026-10-23T20:21:42.170Z"},"provenance":{"textHash":"20e8a5ad0979fbd57a70f25b90961fcf60871f22327de8b3180b7de5132bd4c3","observationId":"d57081a018d93a3d60a867805c3895ea56cc6e912cbf583468dbfe1a0ee4f9e9","retrievedAt":"2026-10-23T20:23:37.322Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"98bcba4e-d80b-f514-d8de-f2fee4f5ed74"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"7264790c-0abd-9919-da9d-6aed874798b8"}},"audit":{"status":"passed","reviewedAt":"2026-11-18T19:04:14.562Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/melanie-mitchell-argues-that-benchmark-scores-do-not-establish-61c94d82","formats":{"html":"/readings/melanie-mitchell-argues-that-benchmark-scores-do-not-establish-61c94d82","markdown":"/readings/melanie-mitchell-argues-that-benchmark-scores-do-not-establish-61c94d82.md","json":"/readings/melanie-mitchell-argues-that-benchmark-scores-do-not-establish-61c94d82.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"}],"entities":[{"id":"nature","name":"Nature","kind":"organization","context":"Nature publishes research, news, and commentary across the sciences.","referenceUrls":["https://www.nature.com/","https://www.nature.com/nature.rss"],"path":"/entities/nature","json":"/entities/nature.json","markdown":"/entities/nature.md"},{"id":"ai-now","name":"AI Now Institute","kind":"organization","context":"AI Now Institute studies the power and institutions shaping AI and advocates for public accountability.","referenceUrls":["https://ainowinstitute.org/"],"path":"/entities/ai-now","json":"/entities/ai-now.json","markdown":"/entities/ai-now.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"61c94d82a9d69cce7e211de0bed8efd8cda9aa359743e05d283e7ded341f2a80","revision":"d31de991-524b-3028-4924-5790ab3f61de","revisionKind":"publication-id","title":"Melanie Mitchell argues that benchmark scores do not establish broad human-level abilities","url":"https://escapemeridian.com/readings/melanie-mitchell-argues-that-benchmark-scores-do-not-establish-61c94d82"},"source":{"type":"source","id":"https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks","revision":"20e8a5ad0979fbd57a70f25b90961fcf60871f22327de8b3180b7de5132bd4c3","revisionKind":"sha256-source-text","title":"\"AI now beats humans at basic tasks\": Really?","url":"https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"61c94d82a9d69cce7e211de0bed8efd8cda9aa359743e05d283e7ded341f2a80","title":"Melanie Mitchell argues that benchmark scores do not establish broad human-level abilities","argument":"Melanie Mitchell published this essay on May 2, 2024, challenging a Nature headline that AI now beats humans at basic tasks. Mitchell argues that high scores on benchmarks—tests designed to compare systems—do not establish that AI surpasses people at the broader abilities those tests are named for. She identifies possible data contamination, reliance on familiar patterns, shortcut learning and weak real-world validity as reasons benchmark results can mislead. For example, she describes studies in which systems used image details or correlated keywords rather than the intended reasoning. Mitchell’s point is not that AI has made little progress: she calls that progress astounding. Rather, she says benchmark performance is not necessarily a good predictor of performance in the real world. She argues that better scientific evaluation should control for shortcuts, test robustness across variations in test items and concepts, and assess how systems perform tasks. The essay’s conclusion is that media reporting should not treat success on a benchmark named for a general ability as success at that ability itself. The evidence here does not include the Nature report, graph or studies Mitchell discusses, so their methods and results cannot be assessed from this text alone.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 23, 2026; originally published May 2, 2024. The Nature report, graph and studies Mitchell discusses are not included in the supplied text, so their methods and results cannot be assessed here.","speaker":{"name":"Melanie Mitchell","roleAtSource":"Publisher"},"source":{"title":"\"AI now beats humans at basic tasks\": Really?","publisher":"Melanie Mitchell","format":"Article","url":"https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks","publishedAt":"2024-05-02T21:48:52.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"Nature’s headline said AI now beats humans at basic tasks, according to Mitchell.","excerpt":"Two weeks ago, Nature, one of the world’s most prestigious journals, had this jarring headline:","url":"https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks","relationship":"Source-stated"},{"text":"Mitchell argues that success on a benchmark named for a general ability does not establish success at that ability.","excerpt":"AI surpassing humans on a benchmark that is named after a general ability is not the same as AI surpassing humans on that general ability.","url":"https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks","relationship":"Source-stated"},{"text":"Mitchell says benchmark scores can mislead for reasons including data contamination, reliance on familiar patterns, shortcut learning and test validity.","excerpt":"There are at least four reasons that AI performance on a benchmark can be misleading.","url":"https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks","relationship":"Source-stated"},{"text":"Mitchell calls for better scientific evaluation that controls for shortcuts, tests robustness across variations, and assesses how machines perform tasks.","excerpt":"ones that control for shortcuts, that test robustness over variations on both the form of test items and on the underlying concepts being assessed, along with other ways to assess the mechanisms by which machines are performing tasks.","url":"https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How would evaluation methods that control for shortcuts and test robustness across variations change the assessment of AI systems’ performance on these abilities?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Melanie Mitchell argues that benchmark scores do not establish broad human-level abilities","summary":"Melanie Mitchell published this essay on May 2, 2024, challenging a Nature headline that AI now beats humans at basic tasks. Mitchell argues that high scores on benchmarks—tests designed to compare systems—do not establish that AI surpasses people at the broader abilities those tests are named for. She identifies possible data contamination, reliance on familiar patterns, shortcut learning and weak real-world validity as reasons benchmark results can mislead. For example, she describes studies in which systems used image details or correlated keywords rather than the intended reasoning. Mitchell’s point is not that AI has made little progress: she calls that progress astounding. Rather, she says benchmark performance is not necessarily a good predictor of performance in the real world. She argues that better scientific evaluation should control for shortcuts, test robustness across variations in test items and concepts, and assess how systems perform tasks. The essay’s conclusion is that media reporting should not treat success on a benchmark named for a general ability as success at that ability itself. The evidence here does not include the Nature report, graph or studies Mitchell discusses, so their methods and results cannot be assessed from this text alone.","question":"How would evaluation methods that control for shortcuts and test robustness across variations change the assessment of AI systems’ performance on these abilities?","claims":[{"text":"Nature’s headline said AI now beats humans at basic tasks, according to Mitchell.","excerpt":"Two weeks ago, Nature, one of the world’s most prestigious journals, had this jarring headline:","support":"source-stated"},{"text":"Mitchell argues that success on a benchmark named for a general ability does not establish success at that ability.","excerpt":"AI surpassing humans on a benchmark that is named after a general ability is not the same as AI surpassing humans on that general ability.","support":"source-stated"},{"text":"Mitchell says benchmark scores can mislead for reasons including data contamination, reliance on familiar patterns, shortcut learning and test validity.","excerpt":"There are at least four reasons that AI performance on a benchmark can be misleading.","support":"source-stated"},{"text":"Mitchell calls for better scientific evaluation that controls for shortcuts, tests robustness across variations, and assesses how machines perform tasks.","excerpt":"ones that control for shortcuts, that test robustness over variations on both the form of test items and on the underlying concepts being assessed, along with other ways to assess the mechanisms by which machines are performing tasks.","support":"source-stated"}],"limitations":["Text as retrieved on October 23, 2026; originally published May 2, 2024.","The Nature report, graph and studies Mitchell discusses are not included in the supplied text, so their methods and results cannot be assessed here."],"mediaId":null,"correctionReason":null,"id":"61c94d82a9d69cce7e211de0bed8efd8cda9aa359743e05d283e7ded341f2a80","publicationId":"d31de991-524b-3028-4924-5790ab3f61de","version":1,"sourceUrl":"https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks","sourceTitle":"\"AI now beats humans at basic tasks\": Really?","publisher":"Melanie Mitchell","publishedAt":"2024-05-02T21:48:52.000Z","originalPublished":{"value":"2024-05-02T21:48:52.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-23T20:21:42.773Z","createdAt":"2026-11-18T19:08:09.855Z","historical":true,"entityIds":[],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://aiguide.substack.com/p/ai-now-beats-humans-at-basic-tasks","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"63bd5df7730c5a253850dd0633fa5dc409686e050c23c820771dc51a4e7a429f","rubricVersion":"em-backfill-context/1","provider":"vercel","model":"typesafe-ai/jev","reasonCodes":["bridge_admitted_by_model"],"confidence":0.55,"decidedAt":"2026-10-23T20:21:42.170Z"},"provenance":{"textHash":"20e8a5ad0979fbd57a70f25b90961fcf60871f22327de8b3180b7de5132bd4c3","observationId":"d57081a018d93a3d60a867805c3895ea56cc6e912cbf583468dbfe1a0ee4f9e9","retrievedAt":"2026-10-23T20:23:37.322Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"98bcba4e-d80b-f514-d8de-f2fee4f5ed74"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"7264790c-0abd-9919-da9d-6aed874798b8"}},"audit":{"status":"passed","reviewedAt":"2026-11-18T19:04:14.562Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"d31de991-524b-3028-4924-5790ab3f61de","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The page is a single essay by Melanie Mitchell, and the reading attributes its arguments to her. Nature's headline is attributed to Nature as Mitchell reports it, and the claims about studies are framed as her descriptions.","excerpt":"But let me repeat an important mantra:"},{"dimension":"summary","status":"pass","explanation":"The reading leads with Mitchell's challenge to the Nature headline. It explains benchmarks in plain words, and a definition is also supplied. It covers her four reasons, the examples, her acknowledgement of astounding progress, her proposed evaluation methods and her mantra for the media. Data contamination is named without a gloss, but it is one item in a list and does not block understanding. The voice and force of the essay are preserved. The question follows from her call for better evaluation.","excerpt":"AI has made astounding progress, but the assumption that the Nature headline makes is not correct."},{"dimension":"date","status":"pass","explanation":"The reading's date of May 2, 2024 matches the source dateline and the UTC calendar day of originalPublished. The evidence kind is publisher-page, not archive-copy, so no archive disclosure is required. The modified date equals the publication date. There is no hindsight framing and no use of later material.","excerpt":"Melanie Mitchell\nMay 02, 2024"},{"dimension":"support","status":"pass","explanation":"Each claim is source-stated and grounded in the text. These cover the headline, the mantra, the four reasons, the evaluation proposals and the point that benchmark performance is not necessarily a good real-world predictor. The examples of image cues and correlated keywords match the ruler and natural-language-inference studies. The limitation about the absent Nature report and studies is accurate. No benefits or costs are invented.","excerpt":"Again and again, it’s been shown that AI performance on benchmarks is not necessarily a good predictor for AI performance in the real world."},{"dimension":"media","status":"pass","explanation":"No media was selected, and no verified media candidates were supplied. This is appropriate.","excerpt":null}],"createdAt":"2026-11-18T19:04:14.562Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}