# STEMBench scores postdoc-level science problems by each field's own measure

[Canonical reading](https://escapemeridian.com/readings/stembench-scores-postdoc-level-science-problems-by-each-field-s-060cf28f)

The authors argue that scientific problem-solving needs a different kind of evaluation: fields judge whether one published method beats another, rather than whether a model matches an answer key. STEMBench scores twelve problems in high-energy physics, astrophysics and neuroscience using each field's figure of merit. For each problem, it includes two measured anchors: a submission without domain knowledge and an expert reference reproducing a published method. The authors say the interval between them indicates how much of the problem is crossed by scientific insight rather than engineering.

Across eight frontier models, task wins did not necessarily mean domain leadership: one model won five tasks but led no domain mean. In a high-effort comparison, at least one model's mean score exceeded the expert anchor on seven tasks. The other five involved uncertainty estimation, latent and causal inference, and cross-site clinical classification. The authors report numerical scores separately from official-submission validity, treating them as different aspects of performance.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv cs.AI · Article

Original source date: Nov 18, 2026 (2026-11-18).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-19T06:04:04.318Z. First observed: 2026-11-19T05:01:26.794Z. Retrieved: 2026-11-19T06:03:09.348Z.

- [Benchmarking LLM abilities on postdoc-level natural science problems](https://arxiv.org/abs/2611.24321)

## Limits

The reported benchmark covers twelve problems in high-energy physics, astrophysics and neuroscience; the abstract does not establish how its results generalize beyond those problems. The abstract reports comparative scores but does not provide the task-level results or enough methodological detail to assess the measurements independently.


## Open question

How well do STEMBench's measured anchors capture the amount of each problem crossed by scientific insight rather than engineering?


## Source-stated

The authors say STEMBench scores each of its twelve research problems using the figure of merit used in that problem's field, with anchors for a submission without domain knowledge and an expert reference reproducing a published method.

> Each of its twelve research problems in high-energy physics, astrophysics and neuroscience is scored by its own field's figure of merit, and each comes with two measured anchors

- [Supporting source](https://arxiv.org/abs/2611.24321)

## Source-stated

The authors report that task-level wins did not align with domain-level leadership: a model won five tasks but led no domain mean.

> a model that wins five of the twelve tasks leads no domain mean.

- [Supporting source](https://arxiv.org/abs/2611.24321)

## Source-stated

In the high-effort cross-model comparison, the authors report at least one model's mean score exceeded the expert anchor on seven tasks; the remaining five covered specified areas of scientific inference and classification.

> at least one model's mean score exceeds the expert anchor on seven of twelve tasks. The remaining five span uncertainty estimation, latent and causal inference, and cross-site clinical classification.

- [Supporting source](https://arxiv.org/abs/2611.24321)

## Source-stated

The authors report numerical scores and official-submission validity separately, as different aspects of performance.

> Numerical scores and official-submission validity capture different aspects of performance and are reported separately.

- [Supporting source](https://arxiv.org/abs/2611.24321)

## Source and contributor context

Avanesyan, Alina, Boldyrev, Alexey, Deng, Liqun, Derkach, Denis, Godunova, Mary, Hushchyn, Mikhail, Ilvovsky, Dmitry, Knyazeva, Irina, Piontkovskaya, Irina, Polubotko, Dmitry, Ratnikov, Fedor, Trofimova, Ekaterina, Zhang, Zhenhe · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Frontier model

One of the most capable AI models available at a given time; the set shifts as models improve.

- [Definition reference](https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper)

## Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

- [Definition reference](https://arxiv.org/abs/2211.09110)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Figure

Figure develops humanoid robots and publishes its own tests of how they perform physical tasks.

- [Explore this context](/entities/figure)
- [Reference](https://www.figure.ai/news)
- [Reference](https://www.figure.ai/news/helix-2-5-zero-shot-30-home-generalization)

## Editorial provenance

Version 1. Publication ID: 3477a7e8-8f02-4da1-8237-4601062632aa.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-19T06:04:04.318Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 23e8de45f0c4814c1b1051ee8ee2d5dc7e073b7fca1e2939cbca277f68ccdb63. Source text hash: 168d52f216c318428497edb53b8ac6c297bed64a2dca390651589525f530238e.


## Revision and source-check history

### Version 1 — 2026-11-19T06:04:04.318Z

The authors argue that scientific problem-solving needs a different kind of evaluation: fields judge whether one published method beats another, rather than whether a model matches an answer key. STEMBench scores twelve problems in high-energy physics, astrophysics and neuroscience using each field's figure of merit. For each problem, it includes two measured anchors: a submission without domain knowledge and an expert reference reproducing a published method. The authors say the interval between them indicates how much of the problem is crossed by scientific insight rather than engineering.

Across eight frontier models, task wins did not necessarily mean domain leadership: one model won five tasks but led no domain mean. In a high-effort comparison, at least one model's mean score exceeded the expert anchor on seven tasks. The other five involved uncertainty estimation, latent and causal inference, and cross-site clinical classification. The authors report numerical scores separately from official-submission validity, treating them as different aspects of performance.

## Original source

arXiv cs.AI · Article

Original source date: Nov 18, 2026 (2026-11-18).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-19T06:04:04.318Z. First observed: 2026-11-19T05:01:26.794Z. Retrieved: 2026-11-19T06:03:09.348Z.

- [Benchmarking LLM abilities on postdoc-level natural science problems](https://arxiv.org/abs/2611.24321)

## Limits

The reported benchmark covers twelve problems in high-energy physics, astrophysics and neuroscience; the abstract does not establish how its results generalize beyond those problems. The abstract reports comparative scores but does not provide the task-level results or enough methodological detail to assess the measurements independently.


## Open question

How well do STEMBench's measured anchors capture the amount of each problem crossed by scientific insight rather than engineering?


## Source-stated

The authors say STEMBench scores each of its twelve research problems using the figure of merit used in that problem's field, with anchors for a submission without domain knowledge and an expert reference reproducing a published method.

> Each of its twelve research problems in high-energy physics, astrophysics and neuroscience is scored by its own field's figure of merit, and each comes with two measured anchors

- [Supporting source](https://arxiv.org/abs/2611.24321)

## Source-stated

The authors report that task-level wins did not align with domain-level leadership: a model won five tasks but led no domain mean.

> a model that wins five of the twelve tasks leads no domain mean.

- [Supporting source](https://arxiv.org/abs/2611.24321)

## Source-stated

In the high-effort cross-model comparison, the authors report at least one model's mean score exceeded the expert anchor on seven tasks; the remaining five covered specified areas of scientific inference and classification.

> at least one model's mean score exceeds the expert anchor on seven of twelve tasks. The remaining five span uncertainty estimation, latent and causal inference, and cross-site clinical classification.

- [Supporting source](https://arxiv.org/abs/2611.24321)

## Source-stated

The authors report numerical scores and official-submission validity separately, as different aspects of performance.

> Numerical scores and official-submission validity capture different aspects of performance and are reported separately.

- [Supporting source](https://arxiv.org/abs/2611.24321)

## Source and contributor context

Avanesyan, Alina, Boldyrev, Alexey, Deng, Liqun, Derkach, Denis, Godunova, Mary, Hushchyn, Mikhail, Ilvovsky, Dmitry, Knyazeva, Irina, Piontkovskaya, Irina, Polubotko, Dmitry, Ratnikov, Fedor, Trofimova, Ekaterina, Zhang, Zhenhe · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Frontier model

One of the most capable AI models available at a given time; the set shifts as models improve.

- [Definition reference](https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper)

## Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

- [Definition reference](https://arxiv.org/abs/2211.09110)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Figure

Figure develops humanoid robots and publishes its own tests of how they perform physical tasks.

- [Explore this context](/entities/figure)
- [Reference](https://www.figure.ai/news)
- [Reference](https://www.figure.ai/news/helix-2-5-zero-shot-30-home-generalization)

## Editorial provenance

Version 1. Publication ID: 3477a7e8-8f02-4da1-8237-4601062632aa.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-19T06:04:04.318Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 23e8de45f0c4814c1b1051ee8ee2d5dc7e073b7fca1e2939cbca277f68ccdb63. Source text hash: 168d52f216c318428497edb53b8ac6c297bed64a2dca390651589525f530238e.


### uncertain — 2026-11-19T06:04:04.318Z — anthropic/claude-opus-5.5

Publication: 3477a7e8-8f02-4da1-8237-4601062632aa

attribution — pass: The reading consistently attributes the benchmark design, its rationale and the results to the paper's authors, who are named on the arXiv page. It does not present the model-versus-expert results as independently established facts.

> STEMBench adopts that criterion directly.

summary — uncertain: The reading is accurate and preserves the core argument. However, several central terms are not explained for a non-specialist: figure of merit, frontier models, domain mean, high-effort comparison, and official-submission validity. A reader may not see that exceeding the expert anchor means beating a reproduced published method on the field's own score. Smallest repair: briefly gloss figure of merit, either here or through a term-library tooltip, as a field's standard quantitative performance measure. Also clarify that the models are large language models and that the anchors come from running code shipped with the benchmark.

> both obtained by running code shipped with the benchmark

date — pass: The original publication date, 2026-11-18, matches the v1 submission date shown on arXiv. The feed date of 2026-11-19 is later and is correctly not used.

> \[Submitted on 18 Nov 2026\]

support — pass: Each claim matches the abstract, including the twelve problems, the two anchors, the five-task winner that leads no domain mean, the seven-of-twelve result, and the separate reporting of validity. The limitations are appropriately cautious. The question follows from the authors' interval claim without asserting an answer, and the reading does not turn the scores into claims about real-world reliability.

> In the high-effort cross-model comparison, at least one model's mean score exceeds the expert anchor on seven of twelve tasks.

media — pass: The selected media is the verified direct link to the arXiv abstract page for this paper. It is the original source and is the most direct way for a reader to access it.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/060cf28f7d7a259dc9ba4c8b54c4c71645c7c43707026bb70a817f8487992cc9/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/stembench-scores-postdoc-level-science-problems-by-each-field-s-060cf28f.json)
