{"id":"a3c159ddd1082bee6ecb9b8361e3724de158855221f8c5d98c5f7df234df5d4f","title":"arXiv cs.AI: SciResearch-Eval evaluates scientific judgment in experimental agents","argument":"Brian T. Weston, Alondra Torres Contreras and Aldair E. Gongora introduce SciResearch-Eval, a benchmark where agents design experiments to estimate a material’s thermal properties using a simulator that may be incomplete or physically wrong. The authors report that when the simulator is missing physics, most models give wrong values while stating they are 95% sure: their intervals contain the truth in 20–62% of cases, compared with 94% for GPT-6 Astra and 99% for Claude Fable 5.2. Astra and Fable sometimes choose experiments, such as gentle heating, where the flawed simulator remains accurate; Astra can stay calibrated even without naming the cause. Asking for wider intervals does not help, the authors report, while stating the strategy improves interval scores but barely moves coverage. More reasoning substantially improves Claude 5.5 Opus, but requires 13–31 times more reasoning; older and open-weight models gain nothing. Fable 5.2 has the best interval score at default effort, but writes six times as many tokens as Astra and falls short at Astra’s token count. The authors release the environment, graders and harness.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence is the paper’s abstract, which reports results on a held-out split of 93 tasks, 9 models and 2,511 episodes; it does not establish how the findings transfer to other scientific tasks. The abstract describes model results in this benchmark and does not establish performance in real-world laboratory work.","speaker":{"name":"Weston, Brian T., Contreras, Alondra Torres, Gongora, Aldair E.","roleAtSource":"Article author"},"source":{"title":"SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.21541","publishedAt":"2026-11-16","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce a benchmark for agents to design experiments and estimate thermal properties using a simulator that may be flawed.","excerpt":"An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong","url":"https://arxiv.org/abs/2611.21541","relationship":"Source-stated"},{"text":"In tasks where the simulator is missing physics, most models’ stated 95% intervals contain the truth in 20–62% of cases; the abstract reports higher coverage for GPT-6 Astra and Claude Fable 5.2.","excerpt":"their ranges contain the truth in 20-62% of cases, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2.","url":"https://arxiv.org/abs/2611.21541","relationship":"Source-stated"},{"text":"Astra and Fable sometimes design experiments where the flawed simulator is accurate, and Astra can remain calibrated without naming the cause.","excerpt":"Astra and Fable design experiments in which the flawed simulator is accurate, such as gentle heating, which keeps Astra calibrated even when it does not name the cause.","url":"https://arxiv.org/abs/2611.21541","relationship":"Source-stated"},{"text":"The authors report that stating the strategy improves interval scores 1.6–2.5-fold, while coverage barely moves.","excerpt":"stating the strategy improves interval scores 1.6-2.5-fold, replicated on a pre-registered fresh split, but coverage barely moves.","url":"https://arxiv.org/abs/2611.21541","relationship":"Source-stated"},{"text":"At maximum effort, Claude 5.5 Opus improves more than tenfold but uses 13–31 times more reasoning; older and open-weight models gain nothing.","excerpt":"at maximum effort Opus improves more than tenfold and is no longer detectably behind Astra at default, but only with 13-31 times more reasoning; older and open-weight models gain nothing.","url":"https://arxiv.org/abs/2611.21541","relationship":"Source-stated"},{"text":"Fable 5.2 has the best interval score at default effort, but uses six times as many tokens as Astra and falls short at Astra’s token count.","excerpt":"Fable 5.2 is the most accurate model at default effort (interval score 1.9 times better than Astra's) but writes 6 times as many tokens, and at Astra's token count it falls short.","url":"https://arxiv.org/abs/2611.21541","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.21541","checkedAt":"2026-11-17T10:52:06.956Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Would the reported calibration and strategy effects hold on scientific tasks beyond measuring thermal properties with this simulator?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: SciResearch-Eval evaluates scientific judgment in experimental agents","summary":"Brian T. Weston, Alondra Torres Contreras and Aldair E. Gongora introduce SciResearch-Eval, a benchmark where agents design experiments to estimate a material’s thermal properties using a simulator that may be incomplete or physically wrong. The authors report that when the simulator is missing physics, most models give wrong values while stating they are 95% sure: their intervals contain the truth in 20–62% of cases, compared with 94% for GPT-6 Astra and 99% for Claude Fable 5.2. Astra and Fable sometimes choose experiments, such as gentle heating, where the flawed simulator remains accurate; Astra can stay calibrated even without naming the cause. Asking for wider intervals does not help, the authors report, while stating the strategy improves interval scores but barely moves coverage. More reasoning substantially improves Claude 5.5 Opus, but requires 13–31 times more reasoning; older and open-weight models gain nothing. Fable 5.2 has the best interval score at default effort, but writes six times as many tokens as Astra and falls short at Astra’s token count. The authors release the environment, graders and harness.","question":"Would the reported calibration and strategy effects hold on scientific tasks beyond measuring thermal properties with this simulator?","claims":[{"text":"The authors introduce a benchmark for agents to design experiments and estimate thermal properties using a simulator that may be flawed.","excerpt":"An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong","support":"source-stated"},{"text":"In tasks where the simulator is missing physics, most models’ stated 95% intervals contain the truth in 20–62% of cases; the abstract reports higher coverage for GPT-6 Astra and Claude Fable 5.2.","excerpt":"their ranges contain the truth in 20-62% of cases, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2.","support":"source-stated"},{"text":"Astra and Fable sometimes design experiments where the flawed simulator is accurate, and Astra can remain calibrated without naming the cause.","excerpt":"Astra and Fable design experiments in which the flawed simulator is accurate, such as gentle heating, which keeps Astra calibrated even when it does not name the cause.","support":"source-stated"},{"text":"The authors report that stating the strategy improves interval scores 1.6–2.5-fold, while coverage barely moves.","excerpt":"stating the strategy improves interval scores 1.6-2.5-fold, replicated on a pre-registered fresh split, but coverage barely moves.","support":"source-stated"},{"text":"At maximum effort, Claude 5.5 Opus improves more than tenfold but uses 13–31 times more reasoning; older and open-weight models gain nothing.","excerpt":"at maximum effort Opus improves more than tenfold and is no longer detectably behind Astra at default, but only with 13-31 times more reasoning; older and open-weight models gain nothing.","support":"source-stated"},{"text":"Fable 5.2 has the best interval score at default effort, but uses six times as many tokens as Astra and falls short at Astra’s token count.","excerpt":"Fable 5.2 is the most accurate model at default effort (interval score 1.9 times better than Astra's) but writes 6 times as many tokens, and at Astra's token count it falls short.","support":"source-stated"}],"limitations":["The evidence is the paper’s abstract, which reports results on a held-out split of 93 tasks, 9 models and 2,511 episodes; it does not establish how the findings transfer to other scientific tasks.","The abstract describes model results in this benchmark and does not establish performance in real-world laboratory work."],"mediaId":"8697d378cdb22e864f5f2ad720e6be01f963888246832569f92c605d3a0ba179","correctionReason":null,"id":"a3c159ddd1082bee6ecb9b8361e3724de158855221f8c5d98c5f7df234df5d4f","publicationId":"a0b85e0e-4031-46cd-87ba-a3baa305130a","version":1,"sourceUrl":"https://arxiv.org/abs/2611.21541","sourceTitle":"SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-16","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-17T08:30:49.099Z","createdAt":"2026-11-17T09:49:40.026Z","historical":false,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"8697d378cdb22e864f5f2ad720e6be01f963888246832569f92c605d3a0ba179","kind":"article","url":"https://arxiv.org/abs/2611.21541","sourceUrl":"https://arxiv.org/abs/2611.21541","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-17T09:48:33.103Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.21541","checkedAt":"2026-11-17T10:52:06.956Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"8697d378cdb22e864f5f2ad720e6be01f963888246832569f92c605d3a0ba179","confidence":0.71,"provider":"typesafe","model":"jev-1.13.0","inputHash":"241b4baca74bf05d8ac3647e0e3477862b14dc7a45f86fa1d898ecd0d4636a8f","decidedAt":"2026-11-17T09:48:35.645Z"},"provenance":{"textHash":"8c0ee4d7bd3d214c0ef7804c12d275b670e3a5c2885770758ed41f32ea7d5733","observationId":"84514ca799819b48ba92371476d2ee9121c0bd1c938ef8bb756f8297a4798009","retrievedAt":"2026-11-17T09:48:32.717Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"eb7bd00b-01a1-41a4-8069-e9adba2e44a8"},"reviewer":{"model":"gpt-6-luna","runId":"72092e5b-7b40-49ad-87fb-0cf4bc0ebf92"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-17T09:49:40.026Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Weston, Brian T.","url":null,"kind":"unknown"},{"name":"Contreras, Alondra Torres","url":null,"kind":"unknown"},{"name":"Gongora, Aldair E.","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Weston, Brian T.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Weston, Brian T.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21541","locator":"head/meta[name=author|citation_author]"},"assertionId":"c60052747dffb63c0b96d203db3c6af46430052e972b974a503973a18bec63e3:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:52:06.956Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c60052747dffb63c0b96d203db3c6af46430052e972b974a503973a18bec63e3","sourceUrl":"https://arxiv.org/abs/2611.21541","contentSha256":"f061bfcac3dae1a5f4f1e194402c54ca2235d3cc9ed6ffc2d5d739d927720b8f","textHash":"8c0ee4d7bd3d214c0ef7804c12d275b670e3a5c2885770758ed41f32ea7d5733","observedAt":"2026-11-17T10:52:06.956Z"}},{"actor":{"name":"Contreras, Alondra Torres","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Contreras, Alondra Torres","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21541","locator":"head/meta[name=author|citation_author]"},"assertionId":"c60052747dffb63c0b96d203db3c6af46430052e972b974a503973a18bec63e3:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:52:06.956Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c60052747dffb63c0b96d203db3c6af46430052e972b974a503973a18bec63e3","sourceUrl":"https://arxiv.org/abs/2611.21541","contentSha256":"f061bfcac3dae1a5f4f1e194402c54ca2235d3cc9ed6ffc2d5d739d927720b8f","textHash":"8c0ee4d7bd3d214c0ef7804c12d275b670e3a5c2885770758ed41f32ea7d5733","observedAt":"2026-11-17T10:52:06.956Z"}},{"actor":{"name":"Gongora, Aldair E.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gongora, Aldair E.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21541","locator":"head/meta[name=author|citation_author]"},"assertionId":"c60052747dffb63c0b96d203db3c6af46430052e972b974a503973a18bec63e3:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:52:06.956Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c60052747dffb63c0b96d203db3c6af46430052e972b974a503973a18bec63e3","sourceUrl":"https://arxiv.org/abs/2611.21541","contentSha256":"f061bfcac3dae1a5f4f1e194402c54ca2235d3cc9ed6ffc2d5d739d927720b8f","textHash":"8c0ee4d7bd3d214c0ef7804c12d275b670e3a5c2885770758ed41f32ea7d5733","observedAt":"2026-11-17T10:52:06.956Z"}}],"participationCoverage":{"total":3,"returned":3,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.21541","snapshotId":"c60052747dffb63c0b96d203db3c6af46430052e972b974a503973a18bec63e3","contentSha256":"f061bfcac3dae1a5f4f1e194402c54ca2235d3cc9ed6ffc2d5d739d927720b8f","textHash":"8c0ee4d7bd3d214c0ef7804c12d275b670e3a5c2885770758ed41f32ea7d5733","observedAt":"2026-11-17T10:52:06.956Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/a3c159ddd1082bee6ecb9b8361e3724de158855221f8c5d98c5f7df234df5d4f/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-sciresearch-eval-evaluates-scientific-judgment-in-a3c159dd","formats":{"html":"/readings/arxiv-cs-ai-sciresearch-eval-evaluates-scientific-judgment-in-a3c159dd","markdown":"/readings/arxiv-cs-ai-sciresearch-eval-evaluates-scientific-judgment-in-a3c159dd.md","json":"/readings/arxiv-cs-ai-sciresearch-eval-evaluates-scientific-judgment-in-a3c159dd.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-open-weights","name":"Open weights","explanation":"A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.","referenceUrl":"https://opensource.org/ai/open-weights"},{"id":"term-tokens","name":"Model tokens","explanation":"Small chunks of text, often parts of words, that a language model reads and writes; usage and prices are often counted in tokens.","referenceUrl":"https://developers.google.com/machine-learning/glossary#token"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"},{"id":"claude","name":"Claude","kind":"model","context":"Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.","referenceUrls":["https://claude.com/product/overview","https://www.anthropic.com/news","https://www.anthropic.com/research/team/economics"],"path":"/entities/claude","json":"/entities/claude.json","markdown":"/entities/claude.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"a3c159ddd1082bee6ecb9b8361e3724de158855221f8c5d98c5f7df234df5d4f","revision":"a0b85e0e-4031-46cd-87ba-a3baa305130a","revisionKind":"publication-id","title":"arXiv cs.AI: SciResearch-Eval evaluates scientific judgment in experimental agents","url":"https://escapemeridian.com/readings/arxiv-cs-ai-sciresearch-eval-evaluates-scientific-judgment-in-a3c159dd"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.21541","revision":"8c0ee4d7bd3d214c0ef7804c12d275b670e3a5c2885770758ed41f32ea7d5733","revisionKind":"sha256-source-text","title":"SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents","url":"https://arxiv.org/abs/2611.21541"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"a3c159ddd1082bee6ecb9b8361e3724de158855221f8c5d98c5f7df234df5d4f","title":"arXiv cs.AI: SciResearch-Eval evaluates scientific judgment in experimental agents","argument":"Brian T. Weston, Alondra Torres Contreras and Aldair E. Gongora introduce SciResearch-Eval, a benchmark where agents design experiments to estimate a material’s thermal properties using a simulator that may be incomplete or physically wrong. The authors report that when the simulator is missing physics, most models give wrong values while stating they are 95% sure: their intervals contain the truth in 20–62% of cases, compared with 94% for GPT-6 Astra and 99% for Claude Fable 5.2. Astra and Fable sometimes choose experiments, such as gentle heating, where the flawed simulator remains accurate; Astra can stay calibrated even without naming the cause. Asking for wider intervals does not help, the authors report, while stating the strategy improves interval scores but barely moves coverage. More reasoning substantially improves Claude 5.5 Opus, but requires 13–31 times more reasoning; older and open-weight models gain nothing. Fable 5.2 has the best interval score at default effort, but writes six times as many tokens as Astra and falls short at Astra’s token count. The authors release the environment, graders and harness.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence is the paper’s abstract, which reports results on a held-out split of 93 tasks, 9 models and 2,511 episodes; it does not establish how the findings transfer to other scientific tasks. The abstract describes model results in this benchmark and does not establish performance in real-world laboratory work.","speaker":{"name":"Weston, Brian T., Contreras, Alondra Torres, Gongora, Aldair E.","roleAtSource":"Article author"},"source":{"title":"SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.21541","publishedAt":"2026-11-16","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce a benchmark for agents to design experiments and estimate thermal properties using a simulator that may be flawed.","excerpt":"An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong","url":"https://arxiv.org/abs/2611.21541","relationship":"Source-stated"},{"text":"In tasks where the simulator is missing physics, most models’ stated 95% intervals contain the truth in 20–62% of cases; the abstract reports higher coverage for GPT-6 Astra and Claude Fable 5.2.","excerpt":"their ranges contain the truth in 20-62% of cases, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2.","url":"https://arxiv.org/abs/2611.21541","relationship":"Source-stated"},{"text":"Astra and Fable sometimes design experiments where the flawed simulator is accurate, and Astra can remain calibrated without naming the cause.","excerpt":"Astra and Fable design experiments in which the flawed simulator is accurate, such as gentle heating, which keeps Astra calibrated even when it does not name the cause.","url":"https://arxiv.org/abs/2611.21541","relationship":"Source-stated"},{"text":"The authors report that stating the strategy improves interval scores 1.6–2.5-fold, while coverage barely moves.","excerpt":"stating the strategy improves interval scores 1.6-2.5-fold, replicated on a pre-registered fresh split, but coverage barely moves.","url":"https://arxiv.org/abs/2611.21541","relationship":"Source-stated"},{"text":"At maximum effort, Claude 5.5 Opus improves more than tenfold but uses 13–31 times more reasoning; older and open-weight models gain nothing.","excerpt":"at maximum effort Opus improves more than tenfold and is no longer detectably behind Astra at default, but only with 13-31 times more reasoning; older and open-weight models gain nothing.","url":"https://arxiv.org/abs/2611.21541","relationship":"Source-stated"},{"text":"Fable 5.2 has the best interval score at default effort, but uses six times as many tokens as Astra and falls short at Astra’s token count.","excerpt":"Fable 5.2 is the most accurate model at default effort (interval score 1.9 times better than Astra's) but writes 6 times as many tokens, and at Astra's token count it falls short.","url":"https://arxiv.org/abs/2611.21541","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.21541","checkedAt":"2026-11-17T10:52:06.956Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Would the reported calibration and strategy effects hold on scientific tasks beyond measuring thermal properties with this simulator?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: SciResearch-Eval evaluates scientific judgment in experimental agents","summary":"Brian T. Weston, Alondra Torres Contreras and Aldair E. Gongora introduce SciResearch-Eval, a benchmark where agents design experiments to estimate a material’s thermal properties using a simulator that may be incomplete or physically wrong. The authors report that when the simulator is missing physics, most models give wrong values while stating they are 95% sure: their intervals contain the truth in 20–62% of cases, compared with 94% for GPT-6 Astra and 99% for Claude Fable 5.2. Astra and Fable sometimes choose experiments, such as gentle heating, where the flawed simulator remains accurate; Astra can stay calibrated even without naming the cause. Asking for wider intervals does not help, the authors report, while stating the strategy improves interval scores but barely moves coverage. More reasoning substantially improves Claude 5.5 Opus, but requires 13–31 times more reasoning; older and open-weight models gain nothing. Fable 5.2 has the best interval score at default effort, but writes six times as many tokens as Astra and falls short at Astra’s token count. The authors release the environment, graders and harness.","question":"Would the reported calibration and strategy effects hold on scientific tasks beyond measuring thermal properties with this simulator?","claims":[{"text":"The authors introduce a benchmark for agents to design experiments and estimate thermal properties using a simulator that may be flawed.","excerpt":"An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong","support":"source-stated"},{"text":"In tasks where the simulator is missing physics, most models’ stated 95% intervals contain the truth in 20–62% of cases; the abstract reports higher coverage for GPT-6 Astra and Claude Fable 5.2.","excerpt":"their ranges contain the truth in 20-62% of cases, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2.","support":"source-stated"},{"text":"Astra and Fable sometimes design experiments where the flawed simulator is accurate, and Astra can remain calibrated without naming the cause.","excerpt":"Astra and Fable design experiments in which the flawed simulator is accurate, such as gentle heating, which keeps Astra calibrated even when it does not name the cause.","support":"source-stated"},{"text":"The authors report that stating the strategy improves interval scores 1.6–2.5-fold, while coverage barely moves.","excerpt":"stating the strategy improves interval scores 1.6-2.5-fold, replicated on a pre-registered fresh split, but coverage barely moves.","support":"source-stated"},{"text":"At maximum effort, Claude 5.5 Opus improves more than tenfold but uses 13–31 times more reasoning; older and open-weight models gain nothing.","excerpt":"at maximum effort Opus improves more than tenfold and is no longer detectably behind Astra at default, but only with 13-31 times more reasoning; older and open-weight models gain nothing.","support":"source-stated"},{"text":"Fable 5.2 has the best interval score at default effort, but uses six times as many tokens as Astra and falls short at Astra’s token count.","excerpt":"Fable 5.2 is the most accurate model at default effort (interval score 1.9 times better than Astra's) but writes 6 times as many tokens, and at Astra's token count it falls short.","support":"source-stated"}],"limitations":["The evidence is the paper’s abstract, which reports results on a held-out split of 93 tasks, 9 models and 2,511 episodes; it does not establish how the findings transfer to other scientific tasks.","The abstract describes model results in this benchmark and does not establish performance in real-world laboratory work."],"mediaId":"8697d378cdb22e864f5f2ad720e6be01f963888246832569f92c605d3a0ba179","correctionReason":null,"id":"a3c159ddd1082bee6ecb9b8361e3724de158855221f8c5d98c5f7df234df5d4f","publicationId":"a0b85e0e-4031-46cd-87ba-a3baa305130a","version":1,"sourceUrl":"https://arxiv.org/abs/2611.21541","sourceTitle":"SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-16","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-17T08:30:49.099Z","createdAt":"2026-11-17T09:49:40.026Z","historical":false,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"8697d378cdb22e864f5f2ad720e6be01f963888246832569f92c605d3a0ba179","kind":"article","url":"https://arxiv.org/abs/2611.21541","sourceUrl":"https://arxiv.org/abs/2611.21541","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-17T09:48:33.103Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.21541","checkedAt":"2026-11-17T10:52:06.956Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"8697d378cdb22e864f5f2ad720e6be01f963888246832569f92c605d3a0ba179","confidence":0.71,"provider":"typesafe","model":"jev-1.13.0","inputHash":"241b4baca74bf05d8ac3647e0e3477862b14dc7a45f86fa1d898ecd0d4636a8f","decidedAt":"2026-11-17T09:48:35.645Z"},"provenance":{"textHash":"8c0ee4d7bd3d214c0ef7804c12d275b670e3a5c2885770758ed41f32ea7d5733","observationId":"84514ca799819b48ba92371476d2ee9121c0bd1c938ef8bb756f8297a4798009","retrievedAt":"2026-11-17T09:48:32.717Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"eb7bd00b-01a1-41a4-8069-e9adba2e44a8"},"reviewer":{"model":"gpt-6-luna","runId":"72092e5b-7b40-49ad-87fb-0cf4bc0ebf92"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-17T09:49:40.026Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Weston, Brian T.","url":null,"kind":"unknown"},{"name":"Contreras, Alondra Torres","url":null,"kind":"unknown"},{"name":"Gongora, Aldair E.","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Weston, Brian T.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Weston, Brian T.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21541","locator":"head/meta[name=author|citation_author]"},"assertionId":"c60052747dffb63c0b96d203db3c6af46430052e972b974a503973a18bec63e3:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:52:06.956Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c60052747dffb63c0b96d203db3c6af46430052e972b974a503973a18bec63e3","sourceUrl":"https://arxiv.org/abs/2611.21541","contentSha256":"f061bfcac3dae1a5f4f1e194402c54ca2235d3cc9ed6ffc2d5d739d927720b8f","textHash":"8c0ee4d7bd3d214c0ef7804c12d275b670e3a5c2885770758ed41f32ea7d5733","observedAt":"2026-11-17T10:52:06.956Z"}},{"actor":{"name":"Contreras, Alondra Torres","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Contreras, Alondra Torres","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21541","locator":"head/meta[name=author|citation_author]"},"assertionId":"c60052747dffb63c0b96d203db3c6af46430052e972b974a503973a18bec63e3:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:52:06.956Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c60052747dffb63c0b96d203db3c6af46430052e972b974a503973a18bec63e3","sourceUrl":"https://arxiv.org/abs/2611.21541","contentSha256":"f061bfcac3dae1a5f4f1e194402c54ca2235d3cc9ed6ffc2d5d739d927720b8f","textHash":"8c0ee4d7bd3d214c0ef7804c12d275b670e3a5c2885770758ed41f32ea7d5733","observedAt":"2026-11-17T10:52:06.956Z"}},{"actor":{"name":"Gongora, Aldair E.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gongora, Aldair E.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21541","locator":"head/meta[name=author|citation_author]"},"assertionId":"c60052747dffb63c0b96d203db3c6af46430052e972b974a503973a18bec63e3:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-17T10:52:06.956Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c60052747dffb63c0b96d203db3c6af46430052e972b974a503973a18bec63e3","sourceUrl":"https://arxiv.org/abs/2611.21541","contentSha256":"f061bfcac3dae1a5f4f1e194402c54ca2235d3cc9ed6ffc2d5d739d927720b8f","textHash":"8c0ee4d7bd3d214c0ef7804c12d275b670e3a5c2885770758ed41f32ea7d5733","observedAt":"2026-11-17T10:52:06.956Z"}}],"participationCoverage":{"total":3,"returned":3,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.21541","snapshotId":"c60052747dffb63c0b96d203db3c6af46430052e972b974a503973a18bec63e3","contentSha256":"f061bfcac3dae1a5f4f1e194402c54ca2235d3cc9ed6ffc2d5d739d927720b8f","textHash":"8c0ee4d7bd3d214c0ef7804c12d275b670e3a5c2885770758ed41f32ea7d5733","observedAt":"2026-11-17T10:52:06.956Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/a3c159ddd1082bee6ecb9b8361e3724de158855221f8c5d98c5f7df234df5d4f/source"}}}],"audits":[{"publicationId":"a0b85e0e-4031-46cd-87ba-a3baa305130a","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The authors Weston, Torres Contreras and Gongora and the arXiv cs.AI venue match the source. Results are framed as the authors' reports, not as established facts.","excerpt":"Authors:Brian T. Weston, Alondra Torres Contreras, Aldair E. Gongora"},{"dimension":"summary","status":"uncertain","explanation":"The reading is faithful but hard for non-specialists to follow at a key point. 'Stating the strategy improves interval scores but barely moves coverage' is opaque: it does not say what strategy is stated, and it does not explain how interval score differs from coverage. 'Calibrated' and 'open-weight' are also unexplained. Smallest repair: add term-library tooltips for these terms and a short clause explaining that distinction. Minor point: 'sometimes choose' adds a hedge the abstract lacks. Optional: the omitted findings on procedural traps and diagnosis rates (Opus 35% vs Astra 66%) are not required.","excerpt":"Asking for wider intervals does not help; stating the strategy improves interval scores 1.6-2.5-fold, replicated on a pre-registered fresh split, but coverage barely moves."},{"dimension":"date","status":"pass","explanation":"originalPublished 2026-11-16 matches the v1 submission date. The feed date of Nov 17 is correctly not used.","excerpt":"[Submitted on 16 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"Each claim is directly supported by the abstract. The question about transfer to other tasks follows from the single-task benchmark. The limitations accurately note the scope (93 tasks, 9 models, 2,511 episodes) and the lack of real-world lab evidence. No benefits or costs are invented.","excerpt":"On a held-out split (93 tasks, 9 models, 2,511 episodes), models handle procedural traps."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct arXiv abstract page for this paper, the most direct access to the source.","excerpt":null}],"createdAt":"2026-11-17T09:49:40.026Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}