Escape Meridian

· Nov 22, 2024

METR reports a benchmark of AI agents on ML research engineering tasks

Markdown · JSON · Original source

published RE-Bench on November 22, 2024, comparing people and on research engineering tasks. The gives participants a computer, a scoring function and a fixed budget to work on tasks such as improving code runtime or designing model architectures. METR reports that agents using 3.5 Sonnet and o1-preview scored higher on average than human experts with short time budgets, while human scores improved faster as budgets grew. At a 32-hour budget, the average human score was almost twice that of the best AI agent. METR says the agents cost several times less per hour and suggests better scaffolding and orchestration could improve performance for a given cost. METR also reports that agents often struggled to respond to new information or build on earlier progress. The authors stress that RE-Bench has few environments, clear objectives and faster feedback than much real-world ML research. They say it remains unclear how well research problems can be packaged into small tasks for models to iterate on. METR hopes RE-Bench will help evaluate dangerous levels of autonomous AI R&D capabilities, which it identifies as a capability of concern in AI safety policies.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

METRArticle · metr.orgOpen original source

Original source

· Article

Source details

Original source date: Nov 22, 2024 (2024-11-22T08:00:00.000Z).

Date basis: Date supplied by the publisher’s feed.

Added to Escape Meridian: 2026-11-16T10:58:22.973Z. First observed: 2026-10-18T03:43:19.850Z. Retrieved: 2026-10-23T23:59:37.953Z.

Referenced by

Recorded citations in public readings. A reference alone does not establish agreement or influence.

No public references recorded here yet.

Reference data (JSON)

Limits

Text as retrieved on October 23, 2026; originally published November 22, 2024. notes that the has a small number of environments and that its clear goals and faster feedback differ from real-world research. METR says better scaffolding and orchestration may improve agent performance; the cited results do not establish how much improvement would follow. The results concern benchmark tasks and do not establish agents’ performance across real-world ML R&D.

Open question

How well do RE-Bench results predict agents’ performance on real-world research with less clearly defined goals and slower feedback?

Source-stated

METR released RE-Bench to compare human experts and frontier model agents on machine learning research engineering tasks.

We’re releasing RE-Bench, a new benchmark for measuring the performance of humans and frontier model agents on ML research engineering tasks.

Source-stated

METR reports that agents scored higher on average than human experts with short time budgets, while human scores improved faster at longer budgets.

When both have short total time budgets (e.g. 2 hours), AI agents that use frontier models Claude-3.5-Sonnet (new) and o1-preview score higher than our human experts on average. However, as we consider longer time budgets, human scores improve at a faster rate than AI agents’.

Source-stated

At a 32-hour time budget, the average human score was almost twice that of the best AI agent.

At a 32-hour time budget, the average human score is almost twice that of the best AI agent.

Source-stated

METR reports that the agents cost less per hour than human experts and suggests that better scaffolding and orchestration could improve agent performance for a given cost.

On the other hand, the hourly cost of running these AI agents is several times cheaper than humans. This suggests that better scaffolding and orchestration could increase model performance substantially for a given cost.

Source-stated

METR says agents often struggle to respond to new information or build on progress over time.

The median AI agent attempt (when we don’t do “best-of-k”) makes very little progress in most environments, and we often observe the agents failing to react appropriately to novel information or struggling to build on their progress over time.

Source-stated

METR says real-world ML research often has slower feedback and less clearly defined goals, leaving uncertainty about whether research can be packaged into small tasks for models to iterate on.

However, real-world ML R&D work often has much slower feedback loops and less clearly-defined goals than in our environments, and it’s unclear to what extent research problems can be packaged into small tasks that models can iterate on effectively.

Source-stated

METR presents RE-Bench as a possible tool for evaluations of dangerous levels of autonomous AI R&D capabilities.

We hope RE-Bench and the methodology we’ve developed will be useful in creating evaluations for dangerous levels of autonomous AI R&D capabilities.

Source and contributor context

METR · Publisher

The publisher is identified here. No individual author has been inferred.

Source media

Media retrieval status: unchecked.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

AI agent

An AI system that chooses and carries out actions toward a goal, often using software tools.

Machine learning

Methods that let computers learn patterns from examples instead of following rules written by hand.

METR

METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.

TIME

TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.

Claude

Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.

Editorial provenance

Version 1. Publication ID: 156ced09-5bc4-3d9d-3e04-00f1d44c1906.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-16T10:53:43.858Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge_admitted_by_model.

Observation: 81b12e818d59f4eeea6ab08acd215d5a54c85d83be5bfd727a232bc3b026b71a. Source text hash: e4c59bfca9d4ec2749a352d2bb84f65412b0c263b44a2997d62b59ce1d4aa742.

Revision and source-check history

Version 1 · 2026-11-16T10:58:22.973Z

METR reports a benchmark of AI agents on ML research engineering tasks

published RE-Bench on November 22, 2024, comparing people and on research engineering tasks. The gives participants a computer, a scoring function and a fixed budget to work on tasks such as improving code runtime or designing model architectures. METR reports that agents using 3.5 Sonnet and o1-preview scored higher on average than human experts with short time budgets, while human scores improved faster as budgets grew. At a 32-hour budget, the average human score was almost twice that of the best AI agent. METR says the agents cost several times less per hour and suggests better scaffolding and orchestration could improve performance for a given cost. METR also reports that agents often struggled to respond to new information or build on earlier progress. The authors stress that RE-Bench has few environments, clear objectives and faster feedback than much real-world ML research. They say it remains unclear how well research problems can be packaged into small tasks for models to iterate on. METR hopes RE-Bench will help evaluate dangerous levels of autonomous AI R&D capabilities, which it identifies as a capability of concern in AI safety policies.

Original source

· Article

Source details

Original source date: Nov 22, 2024 (2024-11-22T08:00:00.000Z).

Date basis: Date supplied by the publisher’s feed.

Added to Escape Meridian: 2026-11-16T10:58:22.973Z. First observed: 2026-10-18T03:43:19.850Z. Retrieved: 2026-10-23T23:59:37.953Z.

Limits

Text as retrieved on October 23, 2026; originally published November 22, 2024. notes that the has a small number of environments and that its clear goals and faster feedback differ from real-world research. METR says better scaffolding and orchestration may improve agent performance; the cited results do not establish how much improvement would follow. The results concern benchmark tasks and do not establish agents’ performance across real-world ML R&D.

Open question

How well do RE-Bench results predict agents’ performance on real-world research with less clearly defined goals and slower feedback?

Source-stated

METR released RE-Bench to compare human experts and frontier model agents on machine learning research engineering tasks.

We’re releasing RE-Bench, a new benchmark for measuring the performance of humans and frontier model agents on ML research engineering tasks.

Source-stated

METR reports that agents scored higher on average than human experts with short time budgets, while human scores improved faster at longer budgets.

When both have short total time budgets (e.g. 2 hours), AI agents that use frontier models Claude-3.5-Sonnet (new) and o1-preview score higher than our human experts on average. However, as we consider longer time budgets, human scores improve at a faster rate than AI agents’.

Source-stated

At a 32-hour time budget, the average human score was almost twice that of the best AI agent.

At a 32-hour time budget, the average human score is almost twice that of the best AI agent.

Source-stated

METR reports that the agents cost less per hour than human experts and suggests that better scaffolding and orchestration could improve agent performance for a given cost.

On the other hand, the hourly cost of running these AI agents is several times cheaper than humans. This suggests that better scaffolding and orchestration could increase model performance substantially for a given cost.

Source-stated

METR says agents often struggle to respond to new information or build on progress over time.

The median AI agent attempt (when we don’t do “best-of-k”) makes very little progress in most environments, and we often observe the agents failing to react appropriately to novel information or struggling to build on their progress over time.

Source-stated

METR says real-world ML research often has slower feedback and less clearly defined goals, leaving uncertainty about whether research can be packaged into small tasks for models to iterate on.

However, real-world ML R&D work often has much slower feedback loops and less clearly-defined goals than in our environments, and it’s unclear to what extent research problems can be packaged into small tasks that models can iterate on effectively.

Source-stated

METR presents RE-Bench as a possible tool for evaluations of dangerous levels of autonomous AI R&D capabilities.

We hope RE-Bench and the methodology we’ve developed will be useful in creating evaluations for dangerous levels of autonomous AI R&D capabilities.

Source and contributor context

METR · Publisher

The publisher is identified here. No individual author has been inferred.

Source media

Media retrieval status: unchecked.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

AI agent

An AI system that chooses and carries out actions toward a goal, often using software tools.

Machine learning

Methods that let computers learn patterns from examples instead of following rules written by hand.

METR

METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.

TIME

TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.

Claude

Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.

Editorial provenance

Version 1. Publication ID: 156ced09-5bc4-3d9d-3e04-00f1d44c1906.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-16T10:53:43.858Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge_admitted_by_model.

Observation: 81b12e818d59f4eeea6ab08acd215d5a54c85d83be5bfd727a232bc3b026b71a. Source text hash: e4c59bfca9d4ec2749a352d2bb84f65412b0c263b44a2997d62b59ce1d4aa742.

uncertain · 2026-11-16T10:53:43.858Z · anthropic/claude-opus-5.5

Publication: 156ced09-5bc4-3d9d-3e04-00f1d44c1906

attribution — pass: The reading is a single METR blog post announcing RE-Bench, and the reading attributes it to METR. Results and hopes are kept as METR's reported findings and statements.

We’re releasing RE-Bench, a new benchmark for measuring the performance of humans and frontier model agents on ML research engineering tasks.

summary — uncertain: The summary is faithful and preserves METR's main tension: agents win at 2 hours, humans win at longer budgets, and real-world transfer is uncertain. But 'scaffolding and orchestration' carries the cost-performance argument and is undefined. The source glosses scaffolds as setups that let a model repeatedly run code. Claude 3.5 Sonnet and o1-preview are also not identified as frontier models from Anthropic and OpenAI. Smallest repair: add a term-library entry for scaffolding and a short clause naming the models' developers.

Modular is a very simple baseline scaffolding that just lets the model repeatedly run code and see the results

date — pass: November 22, 2024 matches the UTC calendar day of evidence.originalPublished (2024-11-22T08:00Z) and the URL path. The kind is publisher-page, so no archive disclosure is needed. There is no hindsight framing. METR's expectation of better elicitation and its plans for more environments are reported as hopes or suggestions, not outcomes.

support — pass: Each claim is quoted closely and kept attributed: the short-budget agent advantage, the near-2x human lead at 32 hours, the cheaper hourly cost, the struggles with novel information, the real-world gaps and the dangerous-capability evaluation aim. The limitations correctly say the scaffolding gains are not established. The question follows directly from METR's own stated uncertainty.

it’s unclear to what extent research problems can be packaged into small tasks that models can iterate on effectively

media — pass: No media was selected, and no verified candidates were supplied. This is consistent with the decision record.

Challenge or correct this reading

Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.