PAPERBENCHX tests whether AI can reproduce science across disciplines

PAPERBENCHX tests whether AI can reproduce science across disciplines

The authors of PAPERBENCHX argue that AI should be able to reproduce known scientific results before it is trusted to discover new ones. Their benchmark tests whether agents can turn a scientific paper into an executable experiment, use specialized scientific software, diagnose computations, and produce evidence that supports a claim. The benchmark covers 93 tasks across 12 research areas. To assess scientific validity rather than plausible-looking answers, the authors replay each submission from scratch and score the regenerated evidence against expert-validated criteria. Across 10 frontier agent configurations, the best fully reproduced 14% of tasks under the 90% threshold; mean rubric scores ranged from 27.53% to 59.08%. The authors say these results show a gap between understanding a paper and reliably producing reproducible scientific evidence. The distinction matters because a plausible output or partial score is not, by itself, a successfully reproduced experiment. The benchmark measures performance on its selected tasks and environments; how well those results generalize to other scientific work remains unknown. An open question is whether agents can reproduce a larger share of tasks when tested across other papers, tools, and scientific environments.

Publication: f3b236d7-c748-433d-8500-08d832d4c32e

attribution — pass: The reading attributes the benchmark, its motivating argument and its interpretation of the results to the paper's authors. Results are framed as the authors' reported findings, not as established fact.

These results reveal a large gap between paper understanding and reliably producing reproducible scientific evidence.

summary — uncertain: The reading drops the paper's main contrast: existing reproduction benchmarks mostly cover AI/ML, and this one covers named fields (electromagnetics, photonics, chemistry, materials science, biology, robotics). The title promises 'across disciplines', but the body says only '12 research areas'. Smallest repair: name the fields and add the AI/ML contrast. The headline 14% figure 'under the 90% threshold' is not explained to non-specialists; the reading should say the abstract does not define it. 'Frontier agent configurations' would also benefit from a tooltip.

Existing research-reproduction benchmarks largely focus on AI and machine learning, where experiments share relatively homogeneous software stacks.

date — pass: The displayed original publication date of 2026-10-29 matches the arXiv v1 submission date. It is not the later feed date.

[Submitted on 29 Oct 2026]

support — pass: Every numeric claim matches the abstract exactly: 93 tasks, 12 areas, 10 configurations, 14%, the 90% threshold and the 27.53%–59.08% rubric range. The point that partial scores are not full reproduction follows from the authors' own wording. The generalization question and limitations are framed as unknowns, not assertions.

indicating that substantial partial progress rarely translates into complete scientific reproduction

media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most direct access to the source.