MIT Technology Review reports on HSG’s changing AI benchmark

MIT Technology Review reports on HSG’s changing AI benchmark

MIT Technology Review’s report, published June 23, 2025, describes Xbench, a benchmark developed by HongShan Capital Group (HSG). HSG says it assesses models with both academic tests and tasks based on real-world workflows, and plans to update questions regularly to keep the benchmark current. The report says HSG made part of the question set open-source and released a leaderboard. Xbench’s academic tests include questions in science and research; its professional tasks initially focus on recruitment and marketing. The report describes tasks such as identifying qualified candidates and matching advertisers with creators. These evaluations aim to assess more than answers to fixed tests, but the report does not establish that benchmark performance predicts success in actual work. HSG developed Xbench as an internal tool for assessing potential investments, then expanded it with outside researchers and professionals before making it public. The report notes that the benchmark’s questions and scoring are designed to assess particular abilities, including reasoning steps, research breadth, factual consistency, and admitting when information is insufficient. Whether regular updates and partly private data will make results useful measures of models’ real-world readiness remains an open question.

Publication: 3831e857-9f36-fe28-b06a-022c325b818a

attribution — pass: The reading attributes the account to MIT Technology Review and identifies HongShan Capital Group (HSG) as Xbench's developer. Its descriptions of plans and design stay attributed to the report or to HSG. The interpretive claim is labeled as Escape Meridian's reading.

Xbench, a new benchmark developed by the Chinese venture capital firm HSG, or HongShan Capital Group

summary — pass: The reading names the actor, the development and its two-part design (academic tests and workflow tasks), the open-sourcing, the leaderboard, the origin as an internal investment tool, and the update plan. The reading page defines "benchmark". The question follows from the evidence and is not leading. Omitting the leaderboard rankings and the outside researcher's quote does not distort the source's meaning.

The other is more like a technical interview round for a job, assessing how much real-world economic value a model might deliver.

date — pass: The stated date, June 23, 2025, matches the UTC day of originalPublished. The reading uses none of the later sidebar items in the captured page, such as the AlphaGo anniversary piece. The planned quarterly updates are presented as commitments, not outcomes. The evidence is a publisher page, so no archive disclosure is needed.

The team has committed to updating the test questions once a quarter and to maintain a half-public, half-private data set.

support — pass: The source text supports every claim: the open-source release, the leaderboard, scoring criteria that go beyond the right answer, quarterly updates, the half-private data set, the recruitment and marketing tasks, and the internal investment origin. The reading labels its interpretation and keeps a clear gap between task design and real-world predictive validity. It does not overclaim benefits.

Development of the benchmark at HongShan began in 2022, following ChatGPT’s breakout success, as an internal tool for assessing which models are worth investing in.

media — pass: No media is selected and no verified media candidates were supplied, so nothing needs checking.