{"id":"7c5655f3e2166042d6f34192e854b273378bb7136d659aaaeb5da15eeee79e25","title":"MIT Technology Review’s The Download examines efforts to improve AI evaluation","argument":"The June twenty-fourth, twenty twenty-five edition of MIT Technology Review’s The Download describes what it calls an “AI evaluation crisis”: model scores can rise on exam-style tests while practical performance remains less clear. The newsletter says many now openly admit that AI testing using benchmarks is broken. A benchmark is a set of tests used to compare systems, but the newsletter’s concern is that passing arbitrary tests may not show whether a model can carry out useful tasks.\n\nThe newsletter points to Xbench, a project developed by HongShan Capital Group, as one effort to address the problem. It evaluates models on real-world tasks as well as tests, and is regularly updated in an attempt to keep it useful. That approach makes the piece’s central point concrete: evaluation depends on what is tested, and scores alone may not capture the abilities people care about. The newsletter does not report whether Xbench or the broader efforts have improved real-world assessment; it directs readers to further coverage.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 31, 2026; originally published June 24, 2025. The newsletter does not report whether Xbench or the wider evaluation efforts have improved real-world assessment.","speaker":{"name":"MIT Technology Review","roleAtSource":"Publisher"},"source":{"title":"The Download: Namibia’s hydrogen hopes, and fixing AI evaluation","publisher":"MIT Technology Review","format":"Article","url":"https://www.technologyreview.com/2025/06/24/1119282/the-download-namibias-hydrogen-hopes-and-fixing-ai-evaluation/","publishedAt":"2025-06-24T12:10:00Z","dateBasis":"Date from the publisher’s own records, through its API."},"evidence":[{"text":"The Download says model scores on tests can rise even as practical performance remains less clear.","excerpt":"Every time a company launches a new AI model, its scores show it beating the capabilities of predecessors. On paper, everything appears to be getting better all the time.","url":"https://www.technologyreview.com/2025/06/24/1119282/the-download-namibias-hydrogen-hopes-and-fixing-ai-evaluation/","relationship":"Source-stated"},{"text":"The newsletter reports that many now openly admit that AI testing using benchmarks is broken.","excerpt":"In fact, many now openly admit that the process of testing AI, using sets of exam-style questions called benchmarks, is broken.","url":"https://www.technologyreview.com/2025/06/24/1119282/the-download-namibias-hydrogen-hopes-and-fixing-ai-evaluation/","relationship":"Source-stated"},{"text":"The newsletter says Xbench evaluates models on real-world tasks as well as tests, and is regularly updated.","excerpt":"It evaluates models not only on the ability to pass arbitrary tests, like most other benchmarks, but also on the ability to execute real-world tasks, which is more unusual. It’s also updated on a regular basis to try to keep it evergreen.","url":"https://www.technologyreview.com/2025/06/24/1119282/the-download-namibias-hydrogen-hopes-and-fixing-ai-evaluation/","relationship":"Source-stated"},{"text":"Escape Meridian reads the newsletter as arguing that evaluation depends on what is tested, so scores alone may not capture abilities people care about.","excerpt":"It evaluates models not only on the ability to pass arbitrary tests, like most other benchmarks, but also on the ability to execute real-world tasks","url":"https://www.technologyreview.com/2025/06/24/1119282/the-download-namibias-hydrogen-hopes-and-fixing-ai-evaluation/","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.technologyreview.com/2025/06/24/1119282/the-download-namibias-hydrogen-hopes-and-fixing-ai-evaluation/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well do evaluations based on real-world tasks predict how AI models will perform in practical use?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"MIT Technology Review’s The Download examines efforts to improve AI evaluation","summary":"The June twenty-fourth, twenty twenty-five edition of MIT Technology Review’s The Download describes what it calls an “AI evaluation crisis”: model scores can rise on exam-style tests while practical performance remains less clear. The newsletter says many now openly admit that AI testing using benchmarks is broken. A benchmark is a set of tests used to compare systems, but the newsletter’s concern is that passing arbitrary tests may not show whether a model can carry out useful tasks.\n\nThe newsletter points to Xbench, a project developed by HongShan Capital Group, as one effort to address the problem. It evaluates models on real-world tasks as well as tests, and is regularly updated in an attempt to keep it useful. That approach makes the piece’s central point concrete: evaluation depends on what is tested, and scores alone may not capture the abilities people care about. The newsletter does not report whether Xbench or the broader efforts have improved real-world assessment; it directs readers to further coverage.","question":"How well do evaluations based on real-world tasks predict how AI models will perform in practical use?","claims":[{"text":"The Download says model scores on tests can rise even as practical performance remains less clear.","excerpt":"Every time a company launches a new AI model, its scores show it beating the capabilities of predecessors. On paper, everything appears to be getting better all the time.","support":"source-stated"},{"text":"The newsletter reports that many now openly admit that AI testing using benchmarks is broken.","excerpt":"In fact, many now openly admit that the process of testing AI, using sets of exam-style questions called benchmarks, is broken.","support":"source-stated"},{"text":"The newsletter says Xbench evaluates models on real-world tasks as well as tests, and is regularly updated.","excerpt":"It evaluates models not only on the ability to pass arbitrary tests, like most other benchmarks, but also on the ability to execute real-world tasks, which is more unusual. It’s also updated on a regular basis to try to keep it evergreen.","support":"source-stated"},{"text":"Escape Meridian reads the newsletter as arguing that evaluation depends on what is tested, so scores alone may not capture abilities people care about.","excerpt":"It evaluates models not only on the ability to pass arbitrary tests, like most other benchmarks, but also on the ability to execute real-world tasks","support":"interpretation"}],"limitations":["Text as retrieved on October 31, 2026; originally published June 24, 2025.","The newsletter does not report whether Xbench or the wider evaluation efforts have improved real-world assessment."],"mediaId":null,"correctionReason":null,"id":"7c5655f3e2166042d6f34192e854b273378bb7136d659aaaeb5da15eeee79e25","publicationId":"e3cb3558-ba42-deea-4d73-eea5b1382b91","version":1,"sourceUrl":"https://www.technologyreview.com/2025/06/24/1119282/the-download-namibias-hydrogen-hopes-and-fixing-ai-evaluation/","sourceTitle":"The Download: Namibia’s hydrogen hopes, and fixing AI evaluation","publisher":"MIT Technology Review","publishedAt":"2025-06-24T12:10:00Z","originalPublished":{"value":"2025-06-24T12:10:00Z","precision":"instant","basis":"backfill:api_published"},"firstObservedAt":"2026-10-31T18:20:21.943Z","createdAt":"2026-11-13T09:53:23.479Z","historical":true,"entityIds":[],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.technologyreview.com/2025/06/24/1119282/the-download-namibias-hydrogen-hopes-and-fixing-ai-evaluation/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"d20b06a950e3506169356a3fe8eb4e777ce475e0cc0b00aab01afd96806f2193","rubricVersion":"unrecorded","provider":"anthropic","model":"claude-opus-5-5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-31T18:20:18.368Z"},"provenance":{"textHash":"8ae3bd69b54d0c42013821771abfc1e872527503a79122a070a390101a050757","observationId":"e941097676c50c458db35a4ade9f7527c655e5ea3fb37261d2e0ed4e593ba5a2","retrievedAt":"2026-10-31T18:21:52.700Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"04301f8e-9dd2-3ee0-d1e7-cfd559c07e1d"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"0c57c9ad-b9b9-a9a4-6dfc-bf4b1a723954"}},"audit":{"status":"passed","reviewedAt":"2026-11-13T09:48:51.815Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/mit-technology-review-s-the-download-examines-efforts-to-7c5655f3","formats":{"html":"/readings/mit-technology-review-s-the-download-examines-efforts-to-7c5655f3","markdown":"/readings/mit-technology-review-s-the-download-examines-efforts-to-7c5655f3.md","json":"/readings/mit-technology-review-s-the-download-examines-efforts-to-7c5655f3.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"}],"entities":[]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"7c5655f3e2166042d6f34192e854b273378bb7136d659aaaeb5da15eeee79e25","revision":"e3cb3558-ba42-deea-4d73-eea5b1382b91","revisionKind":"publication-id","title":"MIT Technology Review’s The Download examines efforts to improve AI evaluation","url":"https://escapemeridian.com/readings/mit-technology-review-s-the-download-examines-efforts-to-7c5655f3"},"source":{"type":"source","id":"https://www.technologyreview.com/2025/06/24/1119282/the-download-namibias-hydrogen-hopes-and-fixing-ai-evaluation/","revision":"8ae3bd69b54d0c42013821771abfc1e872527503a79122a070a390101a050757","revisionKind":"sha256-source-text","title":"The Download: Namibia’s hydrogen hopes, and fixing AI evaluation","url":"https://www.technologyreview.com/2025/06/24/1119282/the-download-namibias-hydrogen-hopes-and-fixing-ai-evaluation/"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"7c5655f3e2166042d6f34192e854b273378bb7136d659aaaeb5da15eeee79e25","title":"MIT Technology Review’s The Download examines efforts to improve AI evaluation","argument":"The June twenty-fourth, twenty twenty-five edition of MIT Technology Review’s The Download describes what it calls an “AI evaluation crisis”: model scores can rise on exam-style tests while practical performance remains less clear. The newsletter says many now openly admit that AI testing using benchmarks is broken. A benchmark is a set of tests used to compare systems, but the newsletter’s concern is that passing arbitrary tests may not show whether a model can carry out useful tasks.\n\nThe newsletter points to Xbench, a project developed by HongShan Capital Group, as one effort to address the problem. It evaluates models on real-world tasks as well as tests, and is regularly updated in an attempt to keep it useful. That approach makes the piece’s central point concrete: evaluation depends on what is tested, and scores alone may not capture the abilities people care about. The newsletter does not report whether Xbench or the broader efforts have improved real-world assessment; it directs readers to further coverage.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 31, 2026; originally published June 24, 2025. The newsletter does not report whether Xbench or the wider evaluation efforts have improved real-world assessment.","speaker":{"name":"MIT Technology Review","roleAtSource":"Publisher"},"source":{"title":"The Download: Namibia’s hydrogen hopes, and fixing AI evaluation","publisher":"MIT Technology Review","format":"Article","url":"https://www.technologyreview.com/2025/06/24/1119282/the-download-namibias-hydrogen-hopes-and-fixing-ai-evaluation/","publishedAt":"2025-06-24T12:10:00Z","dateBasis":"Date from the publisher’s own records, through its API."},"evidence":[{"text":"The Download says model scores on tests can rise even as practical performance remains less clear.","excerpt":"Every time a company launches a new AI model, its scores show it beating the capabilities of predecessors. On paper, everything appears to be getting better all the time.","url":"https://www.technologyreview.com/2025/06/24/1119282/the-download-namibias-hydrogen-hopes-and-fixing-ai-evaluation/","relationship":"Source-stated"},{"text":"The newsletter reports that many now openly admit that AI testing using benchmarks is broken.","excerpt":"In fact, many now openly admit that the process of testing AI, using sets of exam-style questions called benchmarks, is broken.","url":"https://www.technologyreview.com/2025/06/24/1119282/the-download-namibias-hydrogen-hopes-and-fixing-ai-evaluation/","relationship":"Source-stated"},{"text":"The newsletter says Xbench evaluates models on real-world tasks as well as tests, and is regularly updated.","excerpt":"It evaluates models not only on the ability to pass arbitrary tests, like most other benchmarks, but also on the ability to execute real-world tasks, which is more unusual. It’s also updated on a regular basis to try to keep it evergreen.","url":"https://www.technologyreview.com/2025/06/24/1119282/the-download-namibias-hydrogen-hopes-and-fixing-ai-evaluation/","relationship":"Source-stated"},{"text":"Escape Meridian reads the newsletter as arguing that evaluation depends on what is tested, so scores alone may not capture abilities people care about.","excerpt":"It evaluates models not only on the ability to pass arbitrary tests, like most other benchmarks, but also on the ability to execute real-world tasks","url":"https://www.technologyreview.com/2025/06/24/1119282/the-download-namibias-hydrogen-hopes-and-fixing-ai-evaluation/","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.technologyreview.com/2025/06/24/1119282/the-download-namibias-hydrogen-hopes-and-fixing-ai-evaluation/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well do evaluations based on real-world tasks predict how AI models will perform in practical use?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"MIT Technology Review’s The Download examines efforts to improve AI evaluation","summary":"The June twenty-fourth, twenty twenty-five edition of MIT Technology Review’s The Download describes what it calls an “AI evaluation crisis”: model scores can rise on exam-style tests while practical performance remains less clear. The newsletter says many now openly admit that AI testing using benchmarks is broken. A benchmark is a set of tests used to compare systems, but the newsletter’s concern is that passing arbitrary tests may not show whether a model can carry out useful tasks.\n\nThe newsletter points to Xbench, a project developed by HongShan Capital Group, as one effort to address the problem. It evaluates models on real-world tasks as well as tests, and is regularly updated in an attempt to keep it useful. That approach makes the piece’s central point concrete: evaluation depends on what is tested, and scores alone may not capture the abilities people care about. The newsletter does not report whether Xbench or the broader efforts have improved real-world assessment; it directs readers to further coverage.","question":"How well do evaluations based on real-world tasks predict how AI models will perform in practical use?","claims":[{"text":"The Download says model scores on tests can rise even as practical performance remains less clear.","excerpt":"Every time a company launches a new AI model, its scores show it beating the capabilities of predecessors. On paper, everything appears to be getting better all the time.","support":"source-stated"},{"text":"The newsletter reports that many now openly admit that AI testing using benchmarks is broken.","excerpt":"In fact, many now openly admit that the process of testing AI, using sets of exam-style questions called benchmarks, is broken.","support":"source-stated"},{"text":"The newsletter says Xbench evaluates models on real-world tasks as well as tests, and is regularly updated.","excerpt":"It evaluates models not only on the ability to pass arbitrary tests, like most other benchmarks, but also on the ability to execute real-world tasks, which is more unusual. It’s also updated on a regular basis to try to keep it evergreen.","support":"source-stated"},{"text":"Escape Meridian reads the newsletter as arguing that evaluation depends on what is tested, so scores alone may not capture abilities people care about.","excerpt":"It evaluates models not only on the ability to pass arbitrary tests, like most other benchmarks, but also on the ability to execute real-world tasks","support":"interpretation"}],"limitations":["Text as retrieved on October 31, 2026; originally published June 24, 2025.","The newsletter does not report whether Xbench or the wider evaluation efforts have improved real-world assessment."],"mediaId":null,"correctionReason":null,"id":"7c5655f3e2166042d6f34192e854b273378bb7136d659aaaeb5da15eeee79e25","publicationId":"e3cb3558-ba42-deea-4d73-eea5b1382b91","version":1,"sourceUrl":"https://www.technologyreview.com/2025/06/24/1119282/the-download-namibias-hydrogen-hopes-and-fixing-ai-evaluation/","sourceTitle":"The Download: Namibia’s hydrogen hopes, and fixing AI evaluation","publisher":"MIT Technology Review","publishedAt":"2025-06-24T12:10:00Z","originalPublished":{"value":"2025-06-24T12:10:00Z","precision":"instant","basis":"backfill:api_published"},"firstObservedAt":"2026-10-31T18:20:21.943Z","createdAt":"2026-11-13T09:53:23.479Z","historical":true,"entityIds":[],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.technologyreview.com/2025/06/24/1119282/the-download-namibias-hydrogen-hopes-and-fixing-ai-evaluation/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"d20b06a950e3506169356a3fe8eb4e777ce475e0cc0b00aab01afd96806f2193","rubricVersion":"unrecorded","provider":"anthropic","model":"claude-opus-5-5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-31T18:20:18.368Z"},"provenance":{"textHash":"8ae3bd69b54d0c42013821771abfc1e872527503a79122a070a390101a050757","observationId":"e941097676c50c458db35a4ade9f7527c655e5ea3fb37261d2e0ed4e593ba5a2","retrievedAt":"2026-10-31T18:21:52.700Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"04301f8e-9dd2-3ee0-d1e7-cfd559c07e1d"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"0c57c9ad-b9b9-a9a4-6dfc-bf4b1a723954"}},"audit":{"status":"passed","reviewedAt":"2026-11-13T09:48:51.815Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"e3cb3558-ba42-deea-4d73-eea5b1382b91","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the material to MIT Technology Review's The Download, a newsletter that comments on stories in its own words, so it counts as one piece. Xbench is correctly attributed to HongShan Capital Group. Naming Caiwei Chen as the section's byline would be optional, not required.","excerpt":"One of them is Xbench, a benchmark project developed by HongShan Capital Group (formerly Sequoia China)."},{"dimension":"summary","status":"pass","explanation":"The reading concentrates on the newsletter's AI evaluation section and represents it faithfully: rising scores on paper, the claim that benchmark testing is broken, and Xbench's real-world tasks and regular updates. It explains benchmark in plain language, and the term is also defined on the page. Its conclusion on unknowns matches the source, which only points readers to further coverage. A non-specialist can follow it.","excerpt":"In practice, it’s not so simple."},{"dimension":"date","status":"pass","explanation":"evidence.originalPublished is 2025-06-24T12:10Z, and the reading states June 24, 2025, which matches. The Deep Dive sidebar contains later headlines, but the reading does not use them. There is no hindsight framing. The kind is publisher-page, so no archive disclosure is needed. Retrieval is noted in the limitations.","excerpt":"This is today's edition of The Download, our weekday newsletter"},{"dimension":"support","status":"pass","explanation":"Each claim is grounded in quoted source text. The Escape Meridian interpretation is explicitly labelled and stays close to the source's contrast between arbitrary tests and real-world tasks. The question raises a genuine unresolved issue and does not presuppose an answer. The reading attaches no unsupported benefits or costs.","excerpt":"In fact, many now openly admit that the process of testing AI, using sets of exam-style questions called benchmarks, is broken."},{"dimension":"media","status":"pass","explanation":"No media candidates were supplied and none was selected. That is consistent with the evidence.","excerpt":null}],"createdAt":"2026-11-13T09:48:51.815Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}