MIT Technology Review examines the AI evaluation crisis

MIT Technology Review examines the AI evaluation crisis

MIT Technology Review’s story was originally published on June 24, 2025. It argues that familiar benchmarks can stop measuring what people want to know: models may be trained for test scores, encounter questions in their training data, or reach a test’s ceiling. The article describes newer approaches, including LiveCodeBench Pro’s difficult programming problems, benchmarks that keep some questions private or change them over time, and Xbench’s combination of technical tests and practical tasks. These efforts offer different views of model performance, not a comprehensive measure of whether a model is “good.” The article also argues that performance is not the only concern: in real-world settings involving money or safety, an unreliable AI agent’s wrong move could have serious consequences. Human preference tests bring another problem, as users may favor agreeable answers even when they are wrong. The story’s conclusion is that no truly comprehensive scoreboard exists; social, emotional, and interdisciplinary dimensions remain hard to assess. Its broader concern is that a race to publish can reward speed and narrow short-term wins over depth and long-term insight. The article leaves open how to evaluate qualities such as creativity, which the writer says have received little research.

Publication: 4068dc48-999f-2dbe-c2eb-a6cc879d034b

attribution — uncertain: Attributing the piece to MIT Technology Review is fine, but two points blur whose argument is whose. The race-to-publish critique is NYU professor Saining Xie's CVPR critique, which the writer finds powerful and only tentatively extends to benchmarks; the summary calls it the article's own "broader concern" without naming Xie. Likewise, the risk point follows a multi-university group's argument that LLM agents should be judged mainly on riskiness. Smallest repair: name Xie in the summary, and say the researchers made the risk argument.

I found his framing powerful—and maybe it applies to benchmarks, too.

summary — pass: The reading leads with the article's argument that benchmarks have gone stale, and it explains the three causes in plain language (teaching to the test, contamination, ceilings). It describes the newer approaches, the risk concern, the flaws of preference testing and the conclusion that no comprehensive scoreboard exists. Benchmark and AI agent are defined on the page. The title is neutral. The question follows from the writer's stated interest in creativity evaluation and from the preference-testing flaw. The writer's closing note that some skepticism is healthy is softened but not erased.

date — pass: The reading states June 24, 2025, which matches the UTC day of originalPublished (2025-06-24T08:50:30Z). The page was modified the same day; the correction note concerns only a model name, which the reading does not use. The related-story sidebar, including a later AlphaGo anniversary piece, is not used. No hindsight framing appears, and Xbench's plans are not presented as outcomes. The kind is publisher-page, so no archive disclosure is needed. The limitations note gives the retrieval date.

support — pass: Each claim matches its excerpt: the reasons benchmarks go stale, the 53%/0% LiveCodeBench Pro figures, SuperGLUE above 90%, little research on evaluating creativity, the money/safety risk, no comprehensive scoreboard, and Xie's quote. These passages are plainly part of the original piece, not the later model-name correction. Describing the newer methods as offering partial views is consistent with the conclusion. The question does not add unsupported assumptions, because the source itself says users may reward flattering but wrong answers.

Users sometimes reward the answer that sounds more flattering or agreeable, even if it’s wrong.

media — pass: No media was selected, and no verified candidates were supplied, so there is no identity or relevance problem.