MIT Technology Review · Jun 24, 2025
MIT Technology Review’s The Download examines efforts to improve AI evaluation
Markdown · JSON · Original source
The June twenty-fourth, twenty twenty-five edition of MIT Technology Review’s The Download describes what it calls an “AI evaluation crisis”: model scores can rise on exam-style tests while practical performance remains less clear. The newsletter says many now openly admit that AI testing using is broken. A benchmark is a set of tests used to compare systems, but the newsletter’s concern is that passing arbitrary tests may not show whether a model can carry out useful tasks. The newsletter points to Xbench, a project developed by HongShan Capital Group, as one effort to address the problem. It evaluates models on real-world tasks as well as tests, and is regularly updated in an attempt to keep it useful. That approach makes the piece’s central point concrete: evaluation depends on what is tested, and scores alone may not capture the abilities people care about. The newsletter does not report whether Xbench or the broader efforts have improved real-world assessment; it directs readers to further coverage.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
MIT Technology Review · Article
Source details
Original source date: Jun 24, 2025 (2025-06-24T12:10:00Z).
Date basis: Date from the publisher’s own records, through its API.
Added to Escape Meridian: 2026-11-13T09:53:23.479Z. First observed: 2026-10-31T18:20:21.943Z. Retrieved: 2026-10-31T18:21:52.700Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
Text as retrieved on October 31, 2026; originally published June 24, 2025. The newsletter does not report whether Xbench or the wider evaluation efforts have improved real-world assessment.
Open question
How well do evaluations based on real-world tasks predict how AI models will perform in practical use?
Source-stated
The Download says model scores on tests can rise even as practical performance remains less clear.
Every time a company launches a new AI model, its scores show it beating the capabilities of predecessors. On paper, everything appears to be getting better all the time.
Source-stated
The newsletter reports that many now openly admit that AI testing using benchmarks is broken.
In fact, many now openly admit that the process of testing AI, using sets of exam-style questions called benchmarks, is broken.
Source-stated
The newsletter says Xbench evaluates models on real-world tasks as well as tests, and is regularly updated.
It evaluates models not only on the ability to pass arbitrary tests, like most other benchmarks, but also on the ability to execute real-world tasks, which is more unusual. It’s also updated on a regular basis to try to keep it evergreen.
Escape Meridian interpretation
Escape Meridian reads the newsletter as arguing that evaluation depends on what is tested, so scores alone may not capture abilities people care about.
It evaluates models not only on the ability to pass arbitrary tests, like most other benchmarks, but also on the ability to execute real-world tasks
Source and contributor context
MIT Technology Review · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Editorial provenance
Version 1. Publication ID: e3cb3558-ba42-deea-4d73-eea5b1382b91.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-13T09:48:51.815Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model, second_opinion_categorical.
Observation: e941097676c50c458db35a4ade9f7527c655e5ea3fb37261d2e0ed4e593ba5a2. Source text hash: 8ae3bd69b54d0c42013821771abfc1e872527503a79122a070a390101a050757.
Revision and source-check history
Version 1 · 2026-11-13T09:53:23.479Z
MIT Technology Review’s The Download examines efforts to improve AI evaluation
The June twenty-fourth, twenty twenty-five edition of MIT Technology Review’s The Download describes what it calls an “AI evaluation crisis”: model scores can rise on exam-style tests while practical performance remains less clear. The newsletter says many now openly admit that AI testing using is broken. A benchmark is a set of tests used to compare systems, but the newsletter’s concern is that passing arbitrary tests may not show whether a model can carry out useful tasks. The newsletter points to Xbench, a project developed by HongShan Capital Group, as one effort to address the problem. It evaluates models on real-world tasks as well as tests, and is regularly updated in an attempt to keep it useful. That approach makes the piece’s central point concrete: evaluation depends on what is tested, and scores alone may not capture the abilities people care about. The newsletter does not report whether Xbench or the broader efforts have improved real-world assessment; it directs readers to further coverage.
Original source
MIT Technology Review · Article
Source details
Original source date: Jun 24, 2025 (2025-06-24T12:10:00Z).
Date basis: Date from the publisher’s own records, through its API.
Added to Escape Meridian: 2026-11-13T09:53:23.479Z. First observed: 2026-10-31T18:20:21.943Z. Retrieved: 2026-10-31T18:21:52.700Z.
Limits
Text as retrieved on October 31, 2026; originally published June 24, 2025. The newsletter does not report whether Xbench or the wider evaluation efforts have improved real-world assessment.
Open question
How well do evaluations based on real-world tasks predict how AI models will perform in practical use?
Source-stated
The Download says model scores on tests can rise even as practical performance remains less clear.
Every time a company launches a new AI model, its scores show it beating the capabilities of predecessors. On paper, everything appears to be getting better all the time.
Source-stated
The newsletter reports that many now openly admit that AI testing using benchmarks is broken.
In fact, many now openly admit that the process of testing AI, using sets of exam-style questions called benchmarks, is broken.
Source-stated
The newsletter says Xbench evaluates models on real-world tasks as well as tests, and is regularly updated.
It evaluates models not only on the ability to pass arbitrary tests, like most other benchmarks, but also on the ability to execute real-world tasks, which is more unusual. It’s also updated on a regular basis to try to keep it evergreen.
Escape Meridian interpretation
Escape Meridian reads the newsletter as arguing that evaluation depends on what is tested, so scores alone may not capture abilities people care about.
It evaluates models not only on the ability to pass arbitrary tests, like most other benchmarks, but also on the ability to execute real-world tasks
Source and contributor context
MIT Technology Review · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Editorial provenance
Version 1. Publication ID: e3cb3558-ba42-deea-4d73-eea5b1382b91.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-13T09:48:51.815Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model, second_opinion_categorical.
Observation: e941097676c50c458db35a4ade9f7527c655e5ea3fb37261d2e0ed4e593ba5a2. Source text hash: 8ae3bd69b54d0c42013821771abfc1e872527503a79122a070a390101a050757.
passed · 2026-11-13T09:48:51.815Z · anthropic/claude-opus-5.5
Publication: e3cb3558-ba42-deea-4d73-eea5b1382b91
attribution — pass: The reading attributes the material to MIT Technology Review's The Download, a newsletter that comments on stories in its own words, so it counts as one piece. Xbench is correctly attributed to HongShan Capital Group. Naming Caiwei Chen as the section's byline would be optional, not required.
One of them is Xbench, a benchmark project developed by HongShan Capital Group (formerly Sequoia China).
summary — pass: The reading concentrates on the newsletter's AI evaluation section and represents it faithfully: rising scores on paper, the claim that benchmark testing is broken, and Xbench's real-world tasks and regular updates. It explains benchmark in plain language, and the term is also defined on the page. Its conclusion on unknowns matches the source, which only points readers to further coverage. A non-specialist can follow it.
In practice, it’s not so simple.
date — pass: evidence.originalPublished is 2025-06-24T12:10Z, and the reading states June 24, 2025, which matches. The Deep Dive sidebar contains later headlines, but the reading does not use them. There is no hindsight framing. The kind is publisher-page, so no archive disclosure is needed. Retrieval is noted in the limitations.
This is today's edition of The Download, our weekday newsletter
support — pass: Each claim is grounded in quoted source text. The Escape Meridian interpretation is explicitly labelled and stays close to the source's contrast between arbitrary tests and real-world tasks. The question raises a genuine unresolved issue and does not presuppose an answer. The reading attaches no unsupported benefits or costs.
In fact, many now openly admit that the process of testing AI, using sets of exam-style questions called benchmarks, is broken.
media — pass: No media candidates were supplied and none was selected. That is consistent with the evidence.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference