# arXiv cs.AI: SciResearch-Eval evaluates scientific judgment in experimental agents

[Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-sciresearch-eval-evaluates-scientific-judgment-in-a3c159dd)

Brian T. Weston, Alondra Torres Contreras and Aldair E. Gongora introduce SciResearch-Eval, a benchmark where agents design experiments to estimate a material’s thermal properties using a simulator that may be incomplete or physically wrong. The authors report that when the simulator is missing physics, most models give wrong values while stating they are 95% sure: their intervals contain the truth in 20–62% of cases, compared with 94% for GPT-6 Astra and 99% for Claude Fable 5.2. Astra and Fable sometimes choose experiments, such as gentle heating, where the flawed simulator remains accurate; Astra can stay calibrated even without naming the cause. Asking for wider intervals does not help, the authors report, while stating the strategy improves interval scores but barely moves coverage. More reasoning substantially improves Claude 5.5 Opus, but requires 13–31 times more reasoning; older and open-weight models gain nothing. Fable 5.2 has the best interval score at default effort, but writes six times as many tokens as Astra and falls short at Astra’s token count. The authors release the environment, graders and harness.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv cs.AI · Article

Original source date: Nov 16, 2026 (2026-11-16).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-17T09:49:40.026Z. First observed: 2026-11-17T08:30:49.099Z. Retrieved: 2026-11-17T09:48:32.717Z.

- [SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents](https://arxiv.org/abs/2611.21541)

## Limits

The evidence is the paper’s abstract, which reports results on a held-out split of 93 tasks, 9 models and 2,511 episodes; it does not establish how the findings transfer to other scientific tasks. The abstract describes model results in this benchmark and does not establish performance in real-world laboratory work.


## Open question

Would the reported calibration and strategy effects hold on scientific tasks beyond measuring thermal properties with this simulator?


## Source-stated

The authors introduce a benchmark for agents to design experiments and estimate thermal properties using a simulator that may be flawed.

> An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong

- [Supporting source](https://arxiv.org/abs/2611.21541)

## Source-stated

In tasks where the simulator is missing physics, most models’ stated 95% intervals contain the truth in 20–62% of cases; the abstract reports higher coverage for GPT-6 Astra and Claude Fable 5.2.

> their ranges contain the truth in 20-62% of cases, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2.

- [Supporting source](https://arxiv.org/abs/2611.21541)

## Source-stated

Astra and Fable sometimes design experiments where the flawed simulator is accurate, and Astra can remain calibrated without naming the cause.

> Astra and Fable design experiments in which the flawed simulator is accurate, such as gentle heating, which keeps Astra calibrated even when it does not name the cause.

- [Supporting source](https://arxiv.org/abs/2611.21541)

## Source-stated

The authors report that stating the strategy improves interval scores 1.6–2.5-fold, while coverage barely moves.

> stating the strategy improves interval scores 1.6-2.5-fold, replicated on a pre-registered fresh split, but coverage barely moves.

- [Supporting source](https://arxiv.org/abs/2611.21541)

## Source-stated

At maximum effort, Claude 5.5 Opus improves more than tenfold but uses 13–31 times more reasoning; older and open-weight models gain nothing.

> at maximum effort Opus improves more than tenfold and is no longer detectably behind Astra at default, but only with 13-31 times more reasoning; older and open-weight models gain nothing.

- [Supporting source](https://arxiv.org/abs/2611.21541)

## Source-stated

Fable 5.2 has the best interval score at default effort, but uses six times as many tokens as Astra and falls short at Astra’s token count.

> Fable 5.2 is the most accurate model at default effort (interval score 1.9 times better than Astra's) but writes 6 times as many tokens, and at Astra's token count it falls short.

- [Supporting source](https://arxiv.org/abs/2611.21541)

## Source and contributor context

Weston, Brian T., Contreras, Alondra Torres, Gongora, Aldair E. · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

- [Definition reference](https://arxiv.org/abs/2211.09110)

## Open weights

A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.

- [Definition reference](https://opensource.org/ai/open-weights)

## Model tokens

Small chunks of text, often parts of words, that a language model reads and writes; usage and prices are often counted in tokens.

- [Definition reference](https://developers.google.com/machine-learning/glossary#token)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Claude

Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.

- [Explore this context](/entities/claude)
- [Reference](https://claude.com/product/overview)
- [Reference](https://www.anthropic.com/news)
- [Reference](https://www.anthropic.com/research/team/economics)

## Editorial provenance

Version 1. Publication ID: a0b85e0e-4031-46cd-87ba-a3baa305130a.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-17T09:49:40.026Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 84514ca799819b48ba92371476d2ee9121c0bd1c938ef8bb756f8297a4798009. Source text hash: 8c0ee4d7bd3d214c0ef7804c12d275b670e3a5c2885770758ed41f32ea7d5733.


## Revision and source-check history

### Version 1 — 2026-11-17T09:49:40.026Z

Brian T. Weston, Alondra Torres Contreras and Aldair E. Gongora introduce SciResearch-Eval, a benchmark where agents design experiments to estimate a material’s thermal properties using a simulator that may be incomplete or physically wrong. The authors report that when the simulator is missing physics, most models give wrong values while stating they are 95% sure: their intervals contain the truth in 20–62% of cases, compared with 94% for GPT-6 Astra and 99% for Claude Fable 5.2. Astra and Fable sometimes choose experiments, such as gentle heating, where the flawed simulator remains accurate; Astra can stay calibrated even without naming the cause. Asking for wider intervals does not help, the authors report, while stating the strategy improves interval scores but barely moves coverage. More reasoning substantially improves Claude 5.5 Opus, but requires 13–31 times more reasoning; older and open-weight models gain nothing. Fable 5.2 has the best interval score at default effort, but writes six times as many tokens as Astra and falls short at Astra’s token count. The authors release the environment, graders and harness.

## Original source

arXiv cs.AI · Article

Original source date: Nov 16, 2026 (2026-11-16).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-17T09:49:40.026Z. First observed: 2026-11-17T08:30:49.099Z. Retrieved: 2026-11-17T09:48:32.717Z.

- [SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents](https://arxiv.org/abs/2611.21541)

## Limits

The evidence is the paper’s abstract, which reports results on a held-out split of 93 tasks, 9 models and 2,511 episodes; it does not establish how the findings transfer to other scientific tasks. The abstract describes model results in this benchmark and does not establish performance in real-world laboratory work.


## Open question

Would the reported calibration and strategy effects hold on scientific tasks beyond measuring thermal properties with this simulator?


## Source-stated

The authors introduce a benchmark for agents to design experiments and estimate thermal properties using a simulator that may be flawed.

> An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong

- [Supporting source](https://arxiv.org/abs/2611.21541)

## Source-stated

In tasks where the simulator is missing physics, most models’ stated 95% intervals contain the truth in 20–62% of cases; the abstract reports higher coverage for GPT-6 Astra and Claude Fable 5.2.

> their ranges contain the truth in 20-62% of cases, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2.

- [Supporting source](https://arxiv.org/abs/2611.21541)

## Source-stated

Astra and Fable sometimes design experiments where the flawed simulator is accurate, and Astra can remain calibrated without naming the cause.

> Astra and Fable design experiments in which the flawed simulator is accurate, such as gentle heating, which keeps Astra calibrated even when it does not name the cause.

- [Supporting source](https://arxiv.org/abs/2611.21541)

## Source-stated

The authors report that stating the strategy improves interval scores 1.6–2.5-fold, while coverage barely moves.

> stating the strategy improves interval scores 1.6-2.5-fold, replicated on a pre-registered fresh split, but coverage barely moves.

- [Supporting source](https://arxiv.org/abs/2611.21541)

## Source-stated

At maximum effort, Claude 5.5 Opus improves more than tenfold but uses 13–31 times more reasoning; older and open-weight models gain nothing.

> at maximum effort Opus improves more than tenfold and is no longer detectably behind Astra at default, but only with 13-31 times more reasoning; older and open-weight models gain nothing.

- [Supporting source](https://arxiv.org/abs/2611.21541)

## Source-stated

Fable 5.2 has the best interval score at default effort, but uses six times as many tokens as Astra and falls short at Astra’s token count.

> Fable 5.2 is the most accurate model at default effort (interval score 1.9 times better than Astra's) but writes 6 times as many tokens, and at Astra's token count it falls short.

- [Supporting source](https://arxiv.org/abs/2611.21541)

## Source and contributor context

Weston, Brian T., Contreras, Alondra Torres, Gongora, Aldair E. · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

- [Definition reference](https://arxiv.org/abs/2211.09110)

## Open weights

A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.

- [Definition reference](https://opensource.org/ai/open-weights)

## Model tokens

Small chunks of text, often parts of words, that a language model reads and writes; usage and prices are often counted in tokens.

- [Definition reference](https://developers.google.com/machine-learning/glossary#token)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Claude

Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.

- [Explore this context](/entities/claude)
- [Reference](https://claude.com/product/overview)
- [Reference](https://www.anthropic.com/news)
- [Reference](https://www.anthropic.com/research/team/economics)

## Editorial provenance

Version 1. Publication ID: a0b85e0e-4031-46cd-87ba-a3baa305130a.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-17T09:49:40.026Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 84514ca799819b48ba92371476d2ee9121c0bd1c938ef8bb756f8297a4798009. Source text hash: 8c0ee4d7bd3d214c0ef7804c12d275b670e3a5c2885770758ed41f32ea7d5733.


### uncertain — 2026-11-17T09:49:40.026Z — anthropic/claude-opus-5.5

Publication: a0b85e0e-4031-46cd-87ba-a3baa305130a

attribution — pass: The authors Weston, Torres Contreras and Gongora and the arXiv cs.AI venue match the source. Results are framed as the authors' reports, not as established facts.

> Authors:Brian T. Weston, Alondra Torres Contreras, Aldair E. Gongora

summary — uncertain: The reading is faithful but hard for non-specialists to follow at a key point. 'Stating the strategy improves interval scores but barely moves coverage' is opaque: it does not say what strategy is stated, and it does not explain how interval score differs from coverage. 'Calibrated' and 'open-weight' are also unexplained. Smallest repair: add term-library tooltips for these terms and a short clause explaining that distinction. Minor point: 'sometimes choose' adds a hedge the abstract lacks. Optional: the omitted findings on procedural traps and diagnosis rates (Opus 35% vs Astra 66%) are not required.

> Asking for wider intervals does not help; stating the strategy improves interval scores 1.6-2.5-fold, replicated on a pre-registered fresh split, but coverage barely moves.

date — pass: originalPublished 2026-11-16 matches the v1 submission date. The feed date of Nov 17 is correctly not used.

> \[Submitted on 16 Nov 2026\]

support — pass: Each claim is directly supported by the abstract. The question about transfer to other tasks follows from the single-task benchmark. The limitations accurately note the scope (93 tasks, 9 models, 2,511 episodes) and the lack of real-world lab evidence. No benefits or costs are invented.

> On a held-out split (93 tasks, 9 models, 2,511 episodes), models handle procedural traps.

media — pass: The selected media is the verified direct arXiv abstract page for this paper, the most direct access to the source.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/a3c159ddd1082bee6ecb9b8361e3724de158855221f8c5d98c5f7df234df5d4f/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-sciresearch-eval-evaluates-scientific-judgment-in-a3c159dd.json)
