{"id":"49f504747c04c545758f036c6d2a2933d25e64cb98ddf32a82d619b43a05b387","title":"PAPERBENCHX tests whether AI can reproduce science across disciplines","argument":"The authors of PAPERBENCHX argue that AI should be able to reproduce known scientific results before it is trusted to discover new ones. Their benchmark tests whether agents can turn a scientific paper into an executable experiment, use specialized scientific software, diagnose computations, and produce evidence that supports a claim.\n\nThe benchmark covers 93 tasks across 12 research areas. To assess scientific validity rather than plausible-looking answers, the authors replay each submission from scratch and score the regenerated evidence against expert-validated criteria. Across 10 frontier agent configurations, the best fully reproduced 14% of tasks under the 90% threshold; mean rubric scores ranged from 27.53% to 59.08%.\n\nThe authors say these results show a gap between understanding a paper and reliably producing reproducible scientific evidence. The distinction matters because a plausible output or partial score is not, by itself, a successfully reproduced experiment. The benchmark measures performance on its selected tasks and environments; how well those results generalize to other scientific work remains unknown.\n\nAn open question is whether agents can reproduce a larger share of tasks when tested across other papers, tools, and scientific environments.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence is the paper's arXiv abstract; it does not provide task-level results, details of the agent configurations, or the full scoring rubric. The reported results concern the benchmark's selected tasks and environments; the abstract does not establish how performance generalizes to other scientific work.","speaker":{"name":"Wu, Ruoyu, Tu, Zengji, Sun, Aolong, Ma, Tianyi, Li, Junren, Xu, Xinbo, Fu, Li, Zhang, Ziyan, Huang, Pinrui, Wang, Xiaochen, Wu, Chengda, Zhou, Yutao, Wu, Jianjun, Chang, Lin, Chang, Baobao, Chen, Liang, Li, Kuan","roleAtSource":"Article author"},"source":{"title":"PAPERBENCHX: Can AI Reproduce Science Across Disciplines?","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.39744","publishedAt":"2026-10-29","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that reproducing known science should be a checkpoint before trusting AI to discover new science.","excerpt":"Before AI can be trusted to discover new science, it should be able to reproduce what science already knows.","url":"https://arxiv.org/abs/2610.39744","relationship":"Source-stated"},{"text":"PAPERBENCHX contains 93 expert-constructed reproduction tasks spanning 12 research areas.","excerpt":"a benchmark of 93 expert-constructed paper-reproduction tasks spanning 12 research areas","url":"https://arxiv.org/abs/2610.39744","relationship":"Source-stated"},{"text":"The benchmark replays submissions from scratch and scores regenerated evidence against expert-validated criteria.","excerpt":"we replay every submission from scratch and score only regenerated evidence against expert-validated criteria","url":"https://arxiv.org/abs/2610.39744","relationship":"Source-stated"},{"text":"Across 10 frontier agent configurations, the best fully reproduces only 14% of tasks under the 90% threshold.","excerpt":"Across 10 frontier agent configurations, the best configuration fully reproduces only 14% tasks under the 90% threshold.","url":"https://arxiv.org/abs/2610.39744","relationship":"Source-stated"},{"text":"The authors report that mean rubric scores ranged from 27.53% to 59.08%, and interpret the results as showing a gap between paper understanding and reliable evidence production.","excerpt":"mean rubric scores range from 27.53% to 59.08%, indicating that substantial partial progress rarely translates into complete scientific reproduction.","url":"https://arxiv.org/abs/2610.39744","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.39744","checkedAt":"2026-11-13T08:54:09.527Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How would agent performance change on papers, tools, and scientific environments outside PAPERBENCHX?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"PAPERBENCHX tests whether AI can reproduce science across disciplines","summary":"The authors of PAPERBENCHX argue that AI should be able to reproduce known scientific results before it is trusted to discover new ones. Their benchmark tests whether agents can turn a scientific paper into an executable experiment, use specialized scientific software, diagnose computations, and produce evidence that supports a claim.\n\nThe benchmark covers 93 tasks across 12 research areas. To assess scientific validity rather than plausible-looking answers, the authors replay each submission from scratch and score the regenerated evidence against expert-validated criteria. Across 10 frontier agent configurations, the best fully reproduced 14% of tasks under the 90% threshold; mean rubric scores ranged from 27.53% to 59.08%.\n\nThe authors say these results show a gap between understanding a paper and reliably producing reproducible scientific evidence. The distinction matters because a plausible output or partial score is not, by itself, a successfully reproduced experiment. The benchmark measures performance on its selected tasks and environments; how well those results generalize to other scientific work remains unknown.\n\nAn open question is whether agents can reproduce a larger share of tasks when tested across other papers, tools, and scientific environments.","question":"How would agent performance change on papers, tools, and scientific environments outside PAPERBENCHX?","claims":[{"text":"The authors argue that reproducing known science should be a checkpoint before trusting AI to discover new science.","excerpt":"Before AI can be trusted to discover new science, it should be able to reproduce what science already knows.","support":"source-stated"},{"text":"PAPERBENCHX contains 93 expert-constructed reproduction tasks spanning 12 research areas.","excerpt":"a benchmark of 93 expert-constructed paper-reproduction tasks spanning 12 research areas","support":"source-stated"},{"text":"The benchmark replays submissions from scratch and scores regenerated evidence against expert-validated criteria.","excerpt":"we replay every submission from scratch and score only regenerated evidence against expert-validated criteria","support":"source-stated"},{"text":"Across 10 frontier agent configurations, the best fully reproduces only 14% of tasks under the 90% threshold.","excerpt":"Across 10 frontier agent configurations, the best configuration fully reproduces only 14% tasks under the 90% threshold.","support":"source-stated"},{"text":"The authors report that mean rubric scores ranged from 27.53% to 59.08%, and interpret the results as showing a gap between paper understanding and reliable evidence production.","excerpt":"mean rubric scores range from 27.53% to 59.08%, indicating that substantial partial progress rarely translates into complete scientific reproduction.","support":"source-stated"}],"limitations":["The evidence is the paper's arXiv abstract; it does not provide task-level results, details of the agent configurations, or the full scoring rubric.","The reported results concern the benchmark's selected tasks and environments; the abstract does not establish how performance generalizes to other scientific work."],"mediaId":"b50b86e0972e0b67755f60f3c46293509529e45e20a4ac82fa4bc3b4c4ec33dd","correctionReason":null,"id":"49f504747c04c545758f036c6d2a2933d25e64cb98ddf32a82d619b43a05b387","publicationId":"f3b236d7-c748-433d-8500-08d832d4c32e","version":1,"sourceUrl":"https://arxiv.org/abs/2610.39744","sourceTitle":"PAPERBENCHX: Can AI Reproduce Science Across Disciplines?","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-29","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-30T06:42:50.120Z","createdAt":"2026-11-13T06:47:20.111Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"b50b86e0972e0b67755f60f3c46293509529e45e20a4ac82fa4bc3b4c4ec33dd","kind":"article","url":"https://arxiv.org/abs/2610.39744","sourceUrl":"https://arxiv.org/abs/2610.39744","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-31T06:03:06.492Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.39744","checkedAt":"2026-11-13T08:54:09.527Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"b50b86e0972e0b67755f60f3c46293509529e45e20a4ac82fa4bc3b4c4ec33dd","confidence":0.91,"provider":"typesafe","model":"jev-1.13.0","inputHash":"e99243b8e11b5b88ac943aa9b27cc9736f2661136fd2c47729c5d86c8f5b3fff","decidedAt":"2026-10-31T06:03:07.280Z"},"provenance":{"textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observationId":"f3df49c12a288891deca3862eb20246e19684085d1e8589a625b9fb8053f2886","retrievedAt":"2026-10-31T06:03:06.385Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"7575b4a9-e63d-4f0c-8e1c-fe27b221a970"},"reviewer":{"model":"gpt-6-luna","runId":"2f732c6b-6b45-455a-8815-a3e886c8744c"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-13T06:47:20.111Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Wu, Ruoyu","url":null,"kind":"unknown"},{"name":"Tu, Zengji","url":null,"kind":"unknown"},{"name":"Sun, Aolong","url":null,"kind":"unknown"},{"name":"Ma, Tianyi","url":null,"kind":"unknown"},{"name":"Li, Junren","url":null,"kind":"unknown"},{"name":"Xu, Xinbo","url":null,"kind":"unknown"},{"name":"Fu, Li","url":null,"kind":"unknown"},{"name":"Zhang, Ziyan","url":null,"kind":"unknown"},{"name":"Huang, Pinrui","url":null,"kind":"unknown"},{"name":"Wang, Xiaochen","url":null,"kind":"unknown"},{"name":"Wu, Chengda","url":null,"kind":"unknown"},{"name":"Zhou, Yutao","url":null,"kind":"unknown"},{"name":"Wu, Jianjun","url":null,"kind":"unknown"},{"name":"Chang, Lin","url":null,"kind":"unknown"},{"name":"Chang, Baobao","url":null,"kind":"unknown"},{"name":"Chen, Liang","url":null,"kind":"unknown"},{"name":"Li, Kuan","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Wu, Ruoyu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wu, Ruoyu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Tu, Zengji","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Tu, Zengji","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Sun, Aolong","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sun, Aolong","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Ma, Tianyi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ma, Tianyi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Li, Junren","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Junren","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Xu, Xinbo","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Xu, Xinbo","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Fu, Li","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Fu, Li","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Zhang, Ziyan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Ziyan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Huang, Pinrui","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Huang, Pinrui","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:8","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Wang, Xiaochen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wang, Xiaochen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:9","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Wu, Chengda","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wu, Chengda","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:10","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Zhou, Yutao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhou, Yutao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:11","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Wu, Jianjun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wu, Jianjun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:12","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Chang, Lin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chang, Lin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:13","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Chang, Baobao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chang, Baobao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:14","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Chen, Liang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chen, Liang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:15","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Li, Kuan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Kuan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:16","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}}],"participationCoverage":{"total":17,"returned":17,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.39744","snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z","basis":"publisher-page-metadata","sourceLinks":55,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/49f504747c04c545758f036c6d2a2933d25e64cb98ddf32a82d619b43a05b387/source"}},"canonicalUrl":"https://escapemeridian.com/readings/paperbenchx-tests-whether-ai-can-reproduce-science-across-49f50474","formats":{"html":"/readings/paperbenchx-tests-whether-ai-can-reproduce-science-across-49f50474","markdown":"/readings/paperbenchx-tests-whether-ai-can-reproduce-science-across-49f50474.md","json":"/readings/paperbenchx-tests-whether-ai-can-reproduce-science-across-49f50474.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"49f504747c04c545758f036c6d2a2933d25e64cb98ddf32a82d619b43a05b387","revision":"f3b236d7-c748-433d-8500-08d832d4c32e","revisionKind":"publication-id","title":"PAPERBENCHX tests whether AI can reproduce science across disciplines","url":"https://escapemeridian.com/readings/paperbenchx-tests-whether-ai-can-reproduce-science-across-49f50474"},"source":{"type":"source","id":"https://arxiv.org/abs/2610.39744","revision":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","revisionKind":"sha256-source-text","title":"PAPERBENCHX: Can AI Reproduce Science Across Disciplines?","url":"https://arxiv.org/abs/2610.39744"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"49f504747c04c545758f036c6d2a2933d25e64cb98ddf32a82d619b43a05b387","title":"PAPERBENCHX tests whether AI can reproduce science across disciplines","argument":"The authors of PAPERBENCHX argue that AI should be able to reproduce known scientific results before it is trusted to discover new ones. Their benchmark tests whether agents can turn a scientific paper into an executable experiment, use specialized scientific software, diagnose computations, and produce evidence that supports a claim.\n\nThe benchmark covers 93 tasks across 12 research areas. To assess scientific validity rather than plausible-looking answers, the authors replay each submission from scratch and score the regenerated evidence against expert-validated criteria. Across 10 frontier agent configurations, the best fully reproduced 14% of tasks under the 90% threshold; mean rubric scores ranged from 27.53% to 59.08%.\n\nThe authors say these results show a gap between understanding a paper and reliably producing reproducible scientific evidence. The distinction matters because a plausible output or partial score is not, by itself, a successfully reproduced experiment. The benchmark measures performance on its selected tasks and environments; how well those results generalize to other scientific work remains unknown.\n\nAn open question is whether agents can reproduce a larger share of tasks when tested across other papers, tools, and scientific environments.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence is the paper's arXiv abstract; it does not provide task-level results, details of the agent configurations, or the full scoring rubric. The reported results concern the benchmark's selected tasks and environments; the abstract does not establish how performance generalizes to other scientific work.","speaker":{"name":"Wu, Ruoyu, Tu, Zengji, Sun, Aolong, Ma, Tianyi, Li, Junren, Xu, Xinbo, Fu, Li, Zhang, Ziyan, Huang, Pinrui, Wang, Xiaochen, Wu, Chengda, Zhou, Yutao, Wu, Jianjun, Chang, Lin, Chang, Baobao, Chen, Liang, Li, Kuan","roleAtSource":"Article author"},"source":{"title":"PAPERBENCHX: Can AI Reproduce Science Across Disciplines?","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.39744","publishedAt":"2026-10-29","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that reproducing known science should be a checkpoint before trusting AI to discover new science.","excerpt":"Before AI can be trusted to discover new science, it should be able to reproduce what science already knows.","url":"https://arxiv.org/abs/2610.39744","relationship":"Source-stated"},{"text":"PAPERBENCHX contains 93 expert-constructed reproduction tasks spanning 12 research areas.","excerpt":"a benchmark of 93 expert-constructed paper-reproduction tasks spanning 12 research areas","url":"https://arxiv.org/abs/2610.39744","relationship":"Source-stated"},{"text":"The benchmark replays submissions from scratch and scores regenerated evidence against expert-validated criteria.","excerpt":"we replay every submission from scratch and score only regenerated evidence against expert-validated criteria","url":"https://arxiv.org/abs/2610.39744","relationship":"Source-stated"},{"text":"Across 10 frontier agent configurations, the best fully reproduces only 14% of tasks under the 90% threshold.","excerpt":"Across 10 frontier agent configurations, the best configuration fully reproduces only 14% tasks under the 90% threshold.","url":"https://arxiv.org/abs/2610.39744","relationship":"Source-stated"},{"text":"The authors report that mean rubric scores ranged from 27.53% to 59.08%, and interpret the results as showing a gap between paper understanding and reliable evidence production.","excerpt":"mean rubric scores range from 27.53% to 59.08%, indicating that substantial partial progress rarely translates into complete scientific reproduction.","url":"https://arxiv.org/abs/2610.39744","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.39744","checkedAt":"2026-11-13T08:54:09.527Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How would agent performance change on papers, tools, and scientific environments outside PAPERBENCHX?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"PAPERBENCHX tests whether AI can reproduce science across disciplines","summary":"The authors of PAPERBENCHX argue that AI should be able to reproduce known scientific results before it is trusted to discover new ones. Their benchmark tests whether agents can turn a scientific paper into an executable experiment, use specialized scientific software, diagnose computations, and produce evidence that supports a claim.\n\nThe benchmark covers 93 tasks across 12 research areas. To assess scientific validity rather than plausible-looking answers, the authors replay each submission from scratch and score the regenerated evidence against expert-validated criteria. Across 10 frontier agent configurations, the best fully reproduced 14% of tasks under the 90% threshold; mean rubric scores ranged from 27.53% to 59.08%.\n\nThe authors say these results show a gap between understanding a paper and reliably producing reproducible scientific evidence. The distinction matters because a plausible output or partial score is not, by itself, a successfully reproduced experiment. The benchmark measures performance on its selected tasks and environments; how well those results generalize to other scientific work remains unknown.\n\nAn open question is whether agents can reproduce a larger share of tasks when tested across other papers, tools, and scientific environments.","question":"How would agent performance change on papers, tools, and scientific environments outside PAPERBENCHX?","claims":[{"text":"The authors argue that reproducing known science should be a checkpoint before trusting AI to discover new science.","excerpt":"Before AI can be trusted to discover new science, it should be able to reproduce what science already knows.","support":"source-stated"},{"text":"PAPERBENCHX contains 93 expert-constructed reproduction tasks spanning 12 research areas.","excerpt":"a benchmark of 93 expert-constructed paper-reproduction tasks spanning 12 research areas","support":"source-stated"},{"text":"The benchmark replays submissions from scratch and scores regenerated evidence against expert-validated criteria.","excerpt":"we replay every submission from scratch and score only regenerated evidence against expert-validated criteria","support":"source-stated"},{"text":"Across 10 frontier agent configurations, the best fully reproduces only 14% of tasks under the 90% threshold.","excerpt":"Across 10 frontier agent configurations, the best configuration fully reproduces only 14% tasks under the 90% threshold.","support":"source-stated"},{"text":"The authors report that mean rubric scores ranged from 27.53% to 59.08%, and interpret the results as showing a gap between paper understanding and reliable evidence production.","excerpt":"mean rubric scores range from 27.53% to 59.08%, indicating that substantial partial progress rarely translates into complete scientific reproduction.","support":"source-stated"}],"limitations":["The evidence is the paper's arXiv abstract; it does not provide task-level results, details of the agent configurations, or the full scoring rubric.","The reported results concern the benchmark's selected tasks and environments; the abstract does not establish how performance generalizes to other scientific work."],"mediaId":"b50b86e0972e0b67755f60f3c46293509529e45e20a4ac82fa4bc3b4c4ec33dd","correctionReason":null,"id":"49f504747c04c545758f036c6d2a2933d25e64cb98ddf32a82d619b43a05b387","publicationId":"f3b236d7-c748-433d-8500-08d832d4c32e","version":1,"sourceUrl":"https://arxiv.org/abs/2610.39744","sourceTitle":"PAPERBENCHX: Can AI Reproduce Science Across Disciplines?","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-29","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-30T06:42:50.120Z","createdAt":"2026-11-13T06:47:20.111Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"b50b86e0972e0b67755f60f3c46293509529e45e20a4ac82fa4bc3b4c4ec33dd","kind":"article","url":"https://arxiv.org/abs/2610.39744","sourceUrl":"https://arxiv.org/abs/2610.39744","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-31T06:03:06.492Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.39744","checkedAt":"2026-11-13T08:54:09.527Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"b50b86e0972e0b67755f60f3c46293509529e45e20a4ac82fa4bc3b4c4ec33dd","confidence":0.91,"provider":"typesafe","model":"jev-1.13.0","inputHash":"e99243b8e11b5b88ac943aa9b27cc9736f2661136fd2c47729c5d86c8f5b3fff","decidedAt":"2026-10-31T06:03:07.280Z"},"provenance":{"textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observationId":"f3df49c12a288891deca3862eb20246e19684085d1e8589a625b9fb8053f2886","retrievedAt":"2026-10-31T06:03:06.385Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"7575b4a9-e63d-4f0c-8e1c-fe27b221a970"},"reviewer":{"model":"gpt-6-luna","runId":"2f732c6b-6b45-455a-8815-a3e886c8744c"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-13T06:47:20.111Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Wu, Ruoyu","url":null,"kind":"unknown"},{"name":"Tu, Zengji","url":null,"kind":"unknown"},{"name":"Sun, Aolong","url":null,"kind":"unknown"},{"name":"Ma, Tianyi","url":null,"kind":"unknown"},{"name":"Li, Junren","url":null,"kind":"unknown"},{"name":"Xu, Xinbo","url":null,"kind":"unknown"},{"name":"Fu, Li","url":null,"kind":"unknown"},{"name":"Zhang, Ziyan","url":null,"kind":"unknown"},{"name":"Huang, Pinrui","url":null,"kind":"unknown"},{"name":"Wang, Xiaochen","url":null,"kind":"unknown"},{"name":"Wu, Chengda","url":null,"kind":"unknown"},{"name":"Zhou, Yutao","url":null,"kind":"unknown"},{"name":"Wu, Jianjun","url":null,"kind":"unknown"},{"name":"Chang, Lin","url":null,"kind":"unknown"},{"name":"Chang, Baobao","url":null,"kind":"unknown"},{"name":"Chen, Liang","url":null,"kind":"unknown"},{"name":"Li, Kuan","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Wu, Ruoyu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wu, Ruoyu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Tu, Zengji","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Tu, Zengji","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Sun, Aolong","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sun, Aolong","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Ma, Tianyi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ma, Tianyi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Li, Junren","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Junren","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Xu, Xinbo","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Xu, Xinbo","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Fu, Li","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Fu, Li","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Zhang, Ziyan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Ziyan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Huang, Pinrui","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Huang, Pinrui","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:8","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Wang, Xiaochen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wang, Xiaochen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:9","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Wu, Chengda","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wu, Chengda","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:10","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Zhou, Yutao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhou, Yutao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:11","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Wu, Jianjun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wu, Jianjun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:12","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Chang, Lin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chang, Lin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:13","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Chang, Baobao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chang, Baobao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:14","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Chen, Liang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chen, Liang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:15","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}},{"actor":{"name":"Li, Kuan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Kuan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.39744","locator":"head/meta[name=author|citation_author]"},"assertionId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06:participation:16","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T08:54:09.527Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","sourceUrl":"https://arxiv.org/abs/2610.39744","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z"}}],"participationCoverage":{"total":17,"returned":17,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.39744","snapshotId":"16e86297764082ed03e613e83111baba8aa75af2d2f8e1e9b867bc2f9f1ced06","contentSha256":"763cf07417f877e3eed7bea1953b0bcc50e18a4586ff3e5a041024c27f0ea3bc","textHash":"3251d79bfe052728e271302c0df70f7bf79490211edb333acf286e4370ab7b83","observedAt":"2026-11-13T08:54:09.527Z","basis":"publisher-page-metadata","sourceLinks":55,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/49f504747c04c545758f036c6d2a2933d25e64cb98ddf32a82d619b43a05b387/source"}}}],"audits":[{"publicationId":"f3b236d7-c748-433d-8500-08d832d4c32e","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the benchmark, its motivating argument and its interpretation of the results to the paper's authors. Results are framed as the authors' reported findings, not as established fact.","excerpt":"These results reveal a large gap between paper understanding and reliably producing reproducible scientific evidence."},{"dimension":"summary","status":"uncertain","explanation":"The reading drops the paper's main contrast: existing reproduction benchmarks mostly cover AI/ML, and this one covers named fields (electromagnetics, photonics, chemistry, materials science, biology, robotics). The title promises 'across disciplines', but the body says only '12 research areas'. Smallest repair: name the fields and add the AI/ML contrast. The headline 14% figure 'under the 90% threshold' is not explained to non-specialists; the reading should say the abstract does not define it. 'Frontier agent configurations' would also benefit from a tooltip.","excerpt":"Existing research-reproduction benchmarks largely focus on AI and machine learning, where experiments share relatively homogeneous software stacks."},{"dimension":"date","status":"pass","explanation":"The displayed original publication date of 2026-10-29 matches the arXiv v1 submission date. It is not the later feed date.","excerpt":"[Submitted on 29 Oct 2026]"},{"dimension":"support","status":"pass","explanation":"Every numeric claim matches the abstract exactly: 93 tasks, 12 areas, 10 configurations, 14%, the 90% threshold and the 27.53%–59.08% rubric range. The point that partial scores are not full reproduction follows from the authors' own wording. The generalization question and limitations are framed as unknowns, not assertions.","excerpt":"indicating that substantial partial progress rarely translates into complete scientific reproduction"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct arXiv abstract page for this paper. It is the most direct access to the source.","excerpt":null}],"createdAt":"2026-11-13T06:47:20.111Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}