{"id":"77da8f6b0a9bba12f88861c644256fd20b7fd732a01ebdd9ae4407ffd2bf3dcc","title":"arXiv cs.AI: NIBench tests autonomous AI agents in neuroimaging","argument":"The authors of NIBench argue that autonomous AI agents’ reliability in neuroimaging needs better evaluation: plausible analyses can still contain errors that affect scientific conclusions. Their benchmark tests agents on workflow execution, learning procedural skills from execution trajectories, and judging scientific claims against raw cohort data. The authors report that leading agents nearly perfectly execute supplied analysis protocols, but performance differences widen when procedural guidance is withheld. Distilled skills improve performance on tasks used to acquire them and transfer between models, while benefits on unseen tasks are inconsistent. Agents can reproduce reported results yet reach conclusions that conflict with expert-derived answers; access to data does not always improve judgment. Escape Meridian reading: the findings distinguish following an analysis protocol from making a scientifically sound judgment. Evaluating agents only on whether they can run analyses may therefore miss errors in what those analyses imply. The abstract reports evaluations of language models within a shared agent framework, but does not provide detailed results for individual models or claims.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence is the paper's abstract; it does not provide detailed results for individual models or claims.","speaker":{"name":"Zhong, Lujia, Xia, Yihao, Zhang, Jianwei, Huang, Shuo, Nie, Xinyu, Xia, Mingyang, Deng, Zhiwei, Yue, Jiaxin, Wang, Xinkai, Zhao, Junhan, Shi, Yonggang","roleAtSource":"Article author"},"source":{"title":"NIBench: benchmarking autonomous AI agents for execution, skill evolution and scientific judgment in neuroimaging","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.15831","publishedAt":"2026-11-11","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors present NIBench as a benchmark for autonomous agents in neuroimaging, evaluating execution, procedural skill evolution, and scientific judgment.","excerpt":"Here we present NIBench, a neuroimaging benchmark comprising three complementary evaluations of autonomous agents.","url":"https://arxiv.org/abs/2611.15831","relationship":"Source-stated"},{"text":"The authors report that leading agents nearly perfectly execute supplied analysis protocols, while withholding procedural guidance widens performance differences.","excerpt":"Leading agents execute supplied analysis protocols nearly perfectly, whereas withholding procedural guidance widens performance differences.","url":"https://arxiv.org/abs/2611.15831","relationship":"Source-stated"},{"text":"The authors report that distilled skills improve performance on acquisition tasks and transfer between models, but gains on unseen tasks are inconsistent.","excerpt":"Distilled skills improve performance on acquisition tasks and transfer between models, but benefits on unseen tasks are inconsistent.","url":"https://arxiv.org/abs/2611.15831","relationship":"Source-stated"},{"text":"The authors report that agents can reproduce reported results yet sometimes reach conclusions inconsistent with expert-derived answers, and that access to data does not always improve judgment.","excerpt":"Agents reproduce their reported results yet sometimes reach conclusions inconsistent with expert-derived answers, and access to data does not always improve judgment.","url":"https://arxiv.org/abs/2611.15831","relationship":"Source-stated"},{"text":"The findings distinguish executing an analysis protocol from reaching a scientifically sound conclusion: success at execution does not by itself establish sound judgment.","excerpt":"Agents reproduce their reported results yet sometimes reach conclusions inconsistent with expert-derived answers","url":"https://arxiv.org/abs/2611.15831","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.15831","checkedAt":"2026-11-13T10:06:09.938Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"What accounts for agents sometimes reaching conclusions inconsistent with expert-derived answers, even when they reproduce reported results?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: NIBench tests autonomous AI agents in neuroimaging","summary":"The authors of NIBench argue that autonomous AI agents’ reliability in neuroimaging needs better evaluation: plausible analyses can still contain errors that affect scientific conclusions. Their benchmark tests agents on workflow execution, learning procedural skills from execution trajectories, and judging scientific claims against raw cohort data. The authors report that leading agents nearly perfectly execute supplied analysis protocols, but performance differences widen when procedural guidance is withheld. Distilled skills improve performance on tasks used to acquire them and transfer between models, while benefits on unseen tasks are inconsistent. Agents can reproduce reported results yet reach conclusions that conflict with expert-derived answers; access to data does not always improve judgment. Escape Meridian reading: the findings distinguish following an analysis protocol from making a scientifically sound judgment. Evaluating agents only on whether they can run analyses may therefore miss errors in what those analyses imply. The abstract reports evaluations of language models within a shared agent framework, but does not provide detailed results for individual models or claims.","question":"What accounts for agents sometimes reaching conclusions inconsistent with expert-derived answers, even when they reproduce reported results?","claims":[{"text":"The authors present NIBench as a benchmark for autonomous agents in neuroimaging, evaluating execution, procedural skill evolution, and scientific judgment.","excerpt":"Here we present NIBench, a neuroimaging benchmark comprising three complementary evaluations of autonomous agents.","support":"source-stated"},{"text":"The authors report that leading agents nearly perfectly execute supplied analysis protocols, while withholding procedural guidance widens performance differences.","excerpt":"Leading agents execute supplied analysis protocols nearly perfectly, whereas withholding procedural guidance widens performance differences.","support":"source-stated"},{"text":"The authors report that distilled skills improve performance on acquisition tasks and transfer between models, but gains on unseen tasks are inconsistent.","excerpt":"Distilled skills improve performance on acquisition tasks and transfer between models, but benefits on unseen tasks are inconsistent.","support":"source-stated"},{"text":"The authors report that agents can reproduce reported results yet sometimes reach conclusions inconsistent with expert-derived answers, and that access to data does not always improve judgment.","excerpt":"Agents reproduce their reported results yet sometimes reach conclusions inconsistent with expert-derived answers, and access to data does not always improve judgment.","support":"source-stated"},{"text":"The findings distinguish executing an analysis protocol from reaching a scientifically sound conclusion: success at execution does not by itself establish sound judgment.","excerpt":"Agents reproduce their reported results yet sometimes reach conclusions inconsistent with expert-derived answers","support":"interpretation"}],"limitations":["The evidence is the paper's abstract; it does not provide detailed results for individual models or claims."],"mediaId":"795590291e1906daf367d32b28bf88825ba60593f8f60354678316faf0ecdb7a","correctionReason":null,"id":"77da8f6b0a9bba12f88861c644256fd20b7fd732a01ebdd9ae4407ffd2bf3dcc","publicationId":"f792980d-9784-4107-8632-7b808a374c7b","version":1,"sourceUrl":"https://arxiv.org/abs/2611.15831","sourceTitle":"NIBench: benchmarking autonomous AI agents for execution, skill evolution and scientific judgment in neuroimaging","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-11","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-13T07:43:38.538Z","createdAt":"2026-11-13T09:03:30.302Z","historical":false,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"795590291e1906daf367d32b28bf88825ba60593f8f60354678316faf0ecdb7a","kind":"article","url":"https://arxiv.org/abs/2611.15831","sourceUrl":"https://arxiv.org/abs/2611.15831","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-13T09:02:31.765Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.15831","checkedAt":"2026-11-13T10:06:09.938Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"795590291e1906daf367d32b28bf88825ba60593f8f60354678316faf0ecdb7a","confidence":0.65,"provider":"typesafe","model":"jev-1.13.0","inputHash":"9b7f0c05f17de429cf37f97ce457867ada16813aebd10fc454e088a0b1e3abb6","decidedAt":"2026-11-13T09:02:33.832Z"},"provenance":{"textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observationId":"db25de89800d204a67ebb60a65f2806c183256d6edd0a71200ef0b7d3d9b9f79","retrievedAt":"2026-11-13T09:02:31.458Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"f1011bcf-10b9-4119-8ffa-85b0d115197b"},"reviewer":{"model":"gpt-6-luna","runId":"d1794526-2957-45ed-8738-28901b012320"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-13T09:03:30.302Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Zhong, Lujia","url":null,"kind":"unknown"},{"name":"Xia, Yihao","url":null,"kind":"unknown"},{"name":"Zhang, Jianwei","url":null,"kind":"unknown"},{"name":"Huang, Shuo","url":null,"kind":"unknown"},{"name":"Nie, Xinyu","url":null,"kind":"unknown"},{"name":"Xia, Mingyang","url":null,"kind":"unknown"},{"name":"Deng, Zhiwei","url":null,"kind":"unknown"},{"name":"Yue, Jiaxin","url":null,"kind":"unknown"},{"name":"Wang, Xinkai","url":null,"kind":"unknown"},{"name":"Zhao, Junhan","url":null,"kind":"unknown"},{"name":"Shi, Yonggang","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Zhong, Lujia","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhong, Lujia","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Xia, Yihao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Xia, Yihao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Zhang, Jianwei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Jianwei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Huang, Shuo","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Huang, Shuo","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Nie, Xinyu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Nie, Xinyu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Xia, Mingyang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Xia, Mingyang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Deng, Zhiwei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Deng, Zhiwei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Yue, Jiaxin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yue, Jiaxin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Wang, Xinkai","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wang, Xinkai","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:8","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Zhao, Junhan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhao, Junhan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:9","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Shi, Yonggang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Shi, Yonggang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:10","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}}],"participationCoverage":{"total":11,"returned":11,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.15831","snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z","basis":"publisher-page-metadata","sourceLinks":49,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/77da8f6b0a9bba12f88861c644256fd20b7fd732a01ebdd9ae4407ffd2bf3dcc/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-nibench-tests-autonomous-ai-agents-in-neuroimaging-77da8f6b","formats":{"html":"/readings/arxiv-cs-ai-nibench-tests-autonomous-ai-agents-in-neuroimaging-77da8f6b","markdown":"/readings/arxiv-cs-ai-nibench-tests-autonomous-ai-agents-in-neuroimaging-77da8f6b.md","json":"/readings/arxiv-cs-ai-nibench-tests-autonomous-ai-agents-in-neuroimaging-77da8f6b.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-ai-agent","name":"AI agent","explanation":"An AI system that chooses and carries out actions toward a goal, often using software tools.","referenceUrl":"https://www.ibm.com/think/topics/ai-agents"},{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"77da8f6b0a9bba12f88861c644256fd20b7fd732a01ebdd9ae4407ffd2bf3dcc","revision":"f792980d-9784-4107-8632-7b808a374c7b","revisionKind":"publication-id","title":"arXiv cs.AI: NIBench tests autonomous AI agents in neuroimaging","url":"https://escapemeridian.com/readings/arxiv-cs-ai-nibench-tests-autonomous-ai-agents-in-neuroimaging-77da8f6b"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.15831","revision":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","revisionKind":"sha256-source-text","title":"NIBench: benchmarking autonomous AI agents for execution, skill evolution and scientific judgment in neuroimaging","url":"https://arxiv.org/abs/2611.15831"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"77da8f6b0a9bba12f88861c644256fd20b7fd732a01ebdd9ae4407ffd2bf3dcc","title":"arXiv cs.AI: NIBench tests autonomous AI agents in neuroimaging","argument":"The authors of NIBench argue that autonomous AI agents’ reliability in neuroimaging needs better evaluation: plausible analyses can still contain errors that affect scientific conclusions. Their benchmark tests agents on workflow execution, learning procedural skills from execution trajectories, and judging scientific claims against raw cohort data. The authors report that leading agents nearly perfectly execute supplied analysis protocols, but performance differences widen when procedural guidance is withheld. Distilled skills improve performance on tasks used to acquire them and transfer between models, while benefits on unseen tasks are inconsistent. Agents can reproduce reported results yet reach conclusions that conflict with expert-derived answers; access to data does not always improve judgment. Escape Meridian reading: the findings distinguish following an analysis protocol from making a scientifically sound judgment. Evaluating agents only on whether they can run analyses may therefore miss errors in what those analyses imply. The abstract reports evaluations of language models within a shared agent framework, but does not provide detailed results for individual models or claims.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence is the paper's abstract; it does not provide detailed results for individual models or claims.","speaker":{"name":"Zhong, Lujia, Xia, Yihao, Zhang, Jianwei, Huang, Shuo, Nie, Xinyu, Xia, Mingyang, Deng, Zhiwei, Yue, Jiaxin, Wang, Xinkai, Zhao, Junhan, Shi, Yonggang","roleAtSource":"Article author"},"source":{"title":"NIBench: benchmarking autonomous AI agents for execution, skill evolution and scientific judgment in neuroimaging","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.15831","publishedAt":"2026-11-11","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors present NIBench as a benchmark for autonomous agents in neuroimaging, evaluating execution, procedural skill evolution, and scientific judgment.","excerpt":"Here we present NIBench, a neuroimaging benchmark comprising three complementary evaluations of autonomous agents.","url":"https://arxiv.org/abs/2611.15831","relationship":"Source-stated"},{"text":"The authors report that leading agents nearly perfectly execute supplied analysis protocols, while withholding procedural guidance widens performance differences.","excerpt":"Leading agents execute supplied analysis protocols nearly perfectly, whereas withholding procedural guidance widens performance differences.","url":"https://arxiv.org/abs/2611.15831","relationship":"Source-stated"},{"text":"The authors report that distilled skills improve performance on acquisition tasks and transfer between models, but gains on unseen tasks are inconsistent.","excerpt":"Distilled skills improve performance on acquisition tasks and transfer between models, but benefits on unseen tasks are inconsistent.","url":"https://arxiv.org/abs/2611.15831","relationship":"Source-stated"},{"text":"The authors report that agents can reproduce reported results yet sometimes reach conclusions inconsistent with expert-derived answers, and that access to data does not always improve judgment.","excerpt":"Agents reproduce their reported results yet sometimes reach conclusions inconsistent with expert-derived answers, and access to data does not always improve judgment.","url":"https://arxiv.org/abs/2611.15831","relationship":"Source-stated"},{"text":"The findings distinguish executing an analysis protocol from reaching a scientifically sound conclusion: success at execution does not by itself establish sound judgment.","excerpt":"Agents reproduce their reported results yet sometimes reach conclusions inconsistent with expert-derived answers","url":"https://arxiv.org/abs/2611.15831","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.15831","checkedAt":"2026-11-13T10:06:09.938Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"What accounts for agents sometimes reaching conclusions inconsistent with expert-derived answers, even when they reproduce reported results?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: NIBench tests autonomous AI agents in neuroimaging","summary":"The authors of NIBench argue that autonomous AI agents’ reliability in neuroimaging needs better evaluation: plausible analyses can still contain errors that affect scientific conclusions. Their benchmark tests agents on workflow execution, learning procedural skills from execution trajectories, and judging scientific claims against raw cohort data. The authors report that leading agents nearly perfectly execute supplied analysis protocols, but performance differences widen when procedural guidance is withheld. Distilled skills improve performance on tasks used to acquire them and transfer between models, while benefits on unseen tasks are inconsistent. Agents can reproduce reported results yet reach conclusions that conflict with expert-derived answers; access to data does not always improve judgment. Escape Meridian reading: the findings distinguish following an analysis protocol from making a scientifically sound judgment. Evaluating agents only on whether they can run analyses may therefore miss errors in what those analyses imply. The abstract reports evaluations of language models within a shared agent framework, but does not provide detailed results for individual models or claims.","question":"What accounts for agents sometimes reaching conclusions inconsistent with expert-derived answers, even when they reproduce reported results?","claims":[{"text":"The authors present NIBench as a benchmark for autonomous agents in neuroimaging, evaluating execution, procedural skill evolution, and scientific judgment.","excerpt":"Here we present NIBench, a neuroimaging benchmark comprising three complementary evaluations of autonomous agents.","support":"source-stated"},{"text":"The authors report that leading agents nearly perfectly execute supplied analysis protocols, while withholding procedural guidance widens performance differences.","excerpt":"Leading agents execute supplied analysis protocols nearly perfectly, whereas withholding procedural guidance widens performance differences.","support":"source-stated"},{"text":"The authors report that distilled skills improve performance on acquisition tasks and transfer between models, but gains on unseen tasks are inconsistent.","excerpt":"Distilled skills improve performance on acquisition tasks and transfer between models, but benefits on unseen tasks are inconsistent.","support":"source-stated"},{"text":"The authors report that agents can reproduce reported results yet sometimes reach conclusions inconsistent with expert-derived answers, and that access to data does not always improve judgment.","excerpt":"Agents reproduce their reported results yet sometimes reach conclusions inconsistent with expert-derived answers, and access to data does not always improve judgment.","support":"source-stated"},{"text":"The findings distinguish executing an analysis protocol from reaching a scientifically sound conclusion: success at execution does not by itself establish sound judgment.","excerpt":"Agents reproduce their reported results yet sometimes reach conclusions inconsistent with expert-derived answers","support":"interpretation"}],"limitations":["The evidence is the paper's abstract; it does not provide detailed results for individual models or claims."],"mediaId":"795590291e1906daf367d32b28bf88825ba60593f8f60354678316faf0ecdb7a","correctionReason":null,"id":"77da8f6b0a9bba12f88861c644256fd20b7fd732a01ebdd9ae4407ffd2bf3dcc","publicationId":"f792980d-9784-4107-8632-7b808a374c7b","version":1,"sourceUrl":"https://arxiv.org/abs/2611.15831","sourceTitle":"NIBench: benchmarking autonomous AI agents for execution, skill evolution and scientific judgment in neuroimaging","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-11","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-13T07:43:38.538Z","createdAt":"2026-11-13T09:03:30.302Z","historical":false,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"795590291e1906daf367d32b28bf88825ba60593f8f60354678316faf0ecdb7a","kind":"article","url":"https://arxiv.org/abs/2611.15831","sourceUrl":"https://arxiv.org/abs/2611.15831","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-13T09:02:31.765Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.15831","checkedAt":"2026-11-13T10:06:09.938Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"795590291e1906daf367d32b28bf88825ba60593f8f60354678316faf0ecdb7a","confidence":0.65,"provider":"typesafe","model":"jev-1.13.0","inputHash":"9b7f0c05f17de429cf37f97ce457867ada16813aebd10fc454e088a0b1e3abb6","decidedAt":"2026-11-13T09:02:33.832Z"},"provenance":{"textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observationId":"db25de89800d204a67ebb60a65f2806c183256d6edd0a71200ef0b7d3d9b9f79","retrievedAt":"2026-11-13T09:02:31.458Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"f1011bcf-10b9-4119-8ffa-85b0d115197b"},"reviewer":{"model":"gpt-6-luna","runId":"d1794526-2957-45ed-8738-28901b012320"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-13T09:03:30.302Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Zhong, Lujia","url":null,"kind":"unknown"},{"name":"Xia, Yihao","url":null,"kind":"unknown"},{"name":"Zhang, Jianwei","url":null,"kind":"unknown"},{"name":"Huang, Shuo","url":null,"kind":"unknown"},{"name":"Nie, Xinyu","url":null,"kind":"unknown"},{"name":"Xia, Mingyang","url":null,"kind":"unknown"},{"name":"Deng, Zhiwei","url":null,"kind":"unknown"},{"name":"Yue, Jiaxin","url":null,"kind":"unknown"},{"name":"Wang, Xinkai","url":null,"kind":"unknown"},{"name":"Zhao, Junhan","url":null,"kind":"unknown"},{"name":"Shi, Yonggang","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Zhong, Lujia","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhong, Lujia","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Xia, Yihao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Xia, Yihao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Zhang, Jianwei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Jianwei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Huang, Shuo","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Huang, Shuo","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Nie, Xinyu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Nie, Xinyu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Xia, Mingyang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Xia, Mingyang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Deng, Zhiwei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Deng, Zhiwei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Yue, Jiaxin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yue, Jiaxin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Wang, Xinkai","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wang, Xinkai","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:8","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Zhao, Junhan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhao, Junhan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:9","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}},{"actor":{"name":"Shi, Yonggang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Shi, Yonggang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.15831","locator":"head/meta[name=author|citation_author]"},"assertionId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268:participation:10","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T10:06:09.938Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","sourceUrl":"https://arxiv.org/abs/2611.15831","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z"}}],"participationCoverage":{"total":11,"returned":11,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.15831","snapshotId":"52d6ef3c1b22c724e2a62cded11b14675410fb35a13dd5cb8ab090768d005268","contentSha256":"200566743c905763a526345e5d44d68067a0e0af32cc2f5d54125d8a6da2a21e","textHash":"7709ee6773279574adf72bb1ab3080d7c052e83d2c5a52ee7b792858faf6559c","observedAt":"2026-11-13T10:06:09.938Z","basis":"publisher-page-metadata","sourceLinks":49,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/77da8f6b0a9bba12f88861c644256fd20b7fd732a01ebdd9ae4407ffd2bf3dcc/source"}}}],"audits":[{"publicationId":"f792980d-9784-4107-8632-7b808a374c7b","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the benchmark, its motivation and its findings to the NIBench authors on arXiv cs.AI. It labels the protocol-versus-judgment point as Escape Meridian interpretation, separate from the authors' claims.","excerpt":"Here we present NIBench, a neuroimaging benchmark comprising three complementary evaluations of autonomous agents."},{"dimension":"summary","status":"uncertain","explanation":"The summary follows the abstract's structure, but three gaps affect understanding. (1) It writes 'reproduce reported results', dropping the source's 'their'. The source means agents reproduce their own reported results, not the published studies' findings. Smallest repair: 'reproduce their own reported results' in the summary and question. (2) 'Distilled skills' and 'execution trajectories' are central terms left unexplained; tooltips are needed. (3) It omits scope that grounds the judgment test: 36 claims from published Alzheimer's disease studies, 19 models, and 56 tasks.","excerpt":"Third, we assess scientific judgment using 36 claims from published Alzheimer's disease studies, requiring agents to analyse raw cohort data and provide reproducible evidence."},{"dimension":"date","status":"pass","explanation":"The originalPublished date of 2026-11-11 matches the arXiv v1 submission date. The feed entry date of 2026-11-13 is correctly not used.","excerpt":"[v1]\nWed, 11 Nov 2026 18:44:56 UTC (1,247 KB)"},{"dimension":"support","status":"uncertain","explanation":"Most claims are verbatim-supported. Claim 4 and the question paraphrase 'reproduce their reported results' as 'reproduce reported results'. This can imply agents replicated the published findings yet still disagreed with experts, which the source does not state. The interpretation claim is suitably labeled and conditional, but it inherits this ambiguity. Restoring 'their (own)' fixes it.","excerpt":"Agents reproduce their reported results yet sometimes reach conclusions inconsistent with expert-derived answers"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified arXiv abstract page itself, which is the direct original source.","excerpt":null}],"createdAt":"2026-11-13T09:03:30.302Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}