arXiv cs.AI: Zihan Dong, Qishi Zhan and Minxuan Hu on Agent Benchmarks

arXiv cs.AI: Zihan Dong, Qishi Zhan and Minxuan Hu on Agent Benchmarks

In their paper, Zihan Dong, Qishi Zhan and Minxuan Hu argue that agent benchmarks can stop distinguishing leading systems because of how results are measured, not because tasks have run out. Many benchmarks reduce a run to pass or fail, which the authors say is too coarse to distinguish top agents when their performance is close. They also say this measure is capped at the reference: on ALE-Bench, the top ten tie when graded against the median human, even though nearly half of runs beat that human. The authors re-score released runs from eight benchmarks and argue that graded credit can preserve differences that pass/fail discards. They derive a clipped log-ratio score with dated anchors, while treating cost as a separate axis. In a Terminal-Bench re-run, models tied by pass/fail differ over tenfold on the quantity discarded by those tests. The paper’s point is that making tasks harder may not fix a measurement that compresses results; whether its proposed scoring approach works beyond the reported re-scoring and re-run remains open.

Publication: b62341aa-9133-46f2-896c-409085f67e95

attribution — pass: The draft correctly attributes the paper and its claims to Zihan Dong, Qishi Zhan and Minxuan Hu on arXiv cs.AI. It keeps the findings framed as the authors' assertions rather than established fact.

Authors:Zihan Dong, Qishi Zhan, Minxuan Hu

summary — uncertain: The summary is faithful, but its central terms are hard to follow without specialist background. It never says these are data-science and ML-engineering agents. It never explains that their tasks produce a measurable quantity, such as a Kaggle metric, contest score or speedup. Without that, 'graded credit' and 'the quantity discarded' are unclear. 'Clipped log-ratio score with dated anchors' is used without a plain-language gloss or tooltip. Smallest repair: add one sentence naming the agent type and the task quantity, and attach term-library tooltips for 'clipped log-ratio score' and 'dated anchors'.

whose tasks yield a quantity (a Kaggle metric, contest score or speedup)

date — pass: The original publication date of 2026-11-10 matches the v1 submission date. The 2026-11-11 feed date is correctly not used.

[Submitted on 10 Nov 2026]

support — pass: Each claim matches the abstract. The draft softens 'most' to 'many' and 'fixes it' to 'can preserve', which is cautious but not contradictory. The closing question and the remark that broader effectiveness remains open are fair, given the abstract-only evidence. No unsupported benefits or costs are implied.

On fixed tasks, a higher bar only moves the problem; graded credit fixes it.

media — pass: The selected media is the verified direct link to the original arXiv abstract page. It is the most direct access to the source.