· Nov 22, 2024
METR reports a benchmark of AI agents on ML research engineering tasks
Markdown · JSON · Original source
published RE-Bench on November 22, 2024, comparing people and on research engineering tasks. The gives participants a computer, a scoring function and a fixed budget to work on tasks such as improving code runtime or designing model architectures. METR reports that agents using 3.5 Sonnet and o1-preview scored higher on average than human experts with short time budgets, while human scores improved faster as budgets grew. At a 32-hour budget, the average human score was almost twice that of the best AI agent. METR says the agents cost several times less per hour and suggests better scaffolding and orchestration could improve performance for a given cost. METR also reports that agents often struggled to respond to new information or build on earlier progress. The authors stress that RE-Bench has few environments, clear objectives and faster feedback than much real-world ML research. They say it remains unclear how well research problems can be packaged into small tasks for models to iterate on. METR hopes RE-Bench will help evaluate dangerous levels of autonomous AI R&D capabilities, which it identifies as a capability of concern in AI safety policies.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
· Article
Source details
Original source date: Nov 22, 2024 (2024-11-22T08:00:00.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-16T10:58:22.973Z. First observed: 2026-10-18T03:43:19.850Z. Retrieved: 2026-10-23T23:59:37.953Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
Text as retrieved on October 23, 2026; originally published November 22, 2024. notes that the has a small number of environments and that its clear goals and faster feedback differ from real-world research. METR says better scaffolding and orchestration may improve agent performance; the cited results do not establish how much improvement would follow. The results concern benchmark tasks and do not establish agents’ performance across real-world ML R&D.
Open question
How well do RE-Bench results predict agents’ performance on real-world research with less clearly defined goals and slower feedback?
Source-stated
METR released RE-Bench to compare human experts and frontier model agents on machine learning research engineering tasks.
We’re releasing RE-Bench, a new benchmark for measuring the performance of humans and frontier model agents on ML research engineering tasks.
Source-stated
METR reports that agents scored higher on average than human experts with short time budgets, while human scores improved faster at longer budgets.
When both have short total time budgets (e.g. 2 hours), AI agents that use frontier models Claude-3.5-Sonnet (new) and o1-preview score higher than our human experts on average. However, as we consider longer time budgets, human scores improve at a faster rate than AI agents’.
Source-stated
At a 32-hour time budget, the average human score was almost twice that of the best AI agent.
At a 32-hour time budget, the average human score is almost twice that of the best AI agent.
Source-stated
METR reports that the agents cost less per hour than human experts and suggests that better scaffolding and orchestration could improve agent performance for a given cost.
On the other hand, the hourly cost of running these AI agents is several times cheaper than humans. This suggests that better scaffolding and orchestration could increase model performance substantially for a given cost.
Source-stated
METR says agents often struggle to respond to new information or build on progress over time.
The median AI agent attempt (when we don’t do “best-of-k”) makes very little progress in most environments, and we often observe the agents failing to react appropriately to novel information or struggling to build on their progress over time.
Source-stated
METR says real-world ML research often has slower feedback and less clearly defined goals, leaving uncertainty about whether research can be packaged into small tasks for models to iterate on.
However, real-world ML R&D work often has much slower feedback loops and less clearly-defined goals than in our environments, and it’s unclear to what extent research problems can be packaged into small tasks that models can iterate on effectively.
Source-stated
METR presents RE-Bench as a possible tool for evaluations of dangerous levels of autonomous AI R&D capabilities.
We hope RE-Bench and the methodology we’ve developed will be useful in creating evaluations for dangerous levels of autonomous AI R&D capabilities.
Source and contributor context
METR · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
AI agent
An AI system that chooses and carries out actions toward a goal, often using software tools.
Machine learning
Methods that let computers learn patterns from examples instead of following rules written by hand.
METR
METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: 156ced09-5bc4-3d9d-3e04-00f1d44c1906.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-16T10:53:43.858Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model.
Observation: 81b12e818d59f4eeea6ab08acd215d5a54c85d83be5bfd727a232bc3b026b71a. Source text hash: e4c59bfca9d4ec2749a352d2bb84f65412b0c263b44a2997d62b59ce1d4aa742.
Revision and source-check history
Version 1 · 2026-11-16T10:58:22.973Z
METR reports a benchmark of AI agents on ML research engineering tasks
published RE-Bench on November 22, 2024, comparing people and on research engineering tasks. The gives participants a computer, a scoring function and a fixed budget to work on tasks such as improving code runtime or designing model architectures. METR reports that agents using 3.5 Sonnet and o1-preview scored higher on average than human experts with short time budgets, while human scores improved faster as budgets grew. At a 32-hour budget, the average human score was almost twice that of the best AI agent. METR says the agents cost several times less per hour and suggests better scaffolding and orchestration could improve performance for a given cost. METR also reports that agents often struggled to respond to new information or build on earlier progress. The authors stress that RE-Bench has few environments, clear objectives and faster feedback than much real-world ML research. They say it remains unclear how well research problems can be packaged into small tasks for models to iterate on. METR hopes RE-Bench will help evaluate dangerous levels of autonomous AI R&D capabilities, which it identifies as a capability of concern in AI safety policies.
Original source
· Article
Source details
Original source date: Nov 22, 2024 (2024-11-22T08:00:00.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-16T10:58:22.973Z. First observed: 2026-10-18T03:43:19.850Z. Retrieved: 2026-10-23T23:59:37.953Z.
Limits
Text as retrieved on October 23, 2026; originally published November 22, 2024. notes that the has a small number of environments and that its clear goals and faster feedback differ from real-world research. METR says better scaffolding and orchestration may improve agent performance; the cited results do not establish how much improvement would follow. The results concern benchmark tasks and do not establish agents’ performance across real-world ML R&D.
Open question
How well do RE-Bench results predict agents’ performance on real-world research with less clearly defined goals and slower feedback?
Source-stated
METR released RE-Bench to compare human experts and frontier model agents on machine learning research engineering tasks.
We’re releasing RE-Bench, a new benchmark for measuring the performance of humans and frontier model agents on ML research engineering tasks.
Source-stated
METR reports that agents scored higher on average than human experts with short time budgets, while human scores improved faster at longer budgets.
When both have short total time budgets (e.g. 2 hours), AI agents that use frontier models Claude-3.5-Sonnet (new) and o1-preview score higher than our human experts on average. However, as we consider longer time budgets, human scores improve at a faster rate than AI agents’.
Source-stated
At a 32-hour time budget, the average human score was almost twice that of the best AI agent.
At a 32-hour time budget, the average human score is almost twice that of the best AI agent.
Source-stated
METR reports that the agents cost less per hour than human experts and suggests that better scaffolding and orchestration could improve agent performance for a given cost.
On the other hand, the hourly cost of running these AI agents is several times cheaper than humans. This suggests that better scaffolding and orchestration could increase model performance substantially for a given cost.
Source-stated
METR says agents often struggle to respond to new information or build on progress over time.
The median AI agent attempt (when we don’t do “best-of-k”) makes very little progress in most environments, and we often observe the agents failing to react appropriately to novel information or struggling to build on their progress over time.
Source-stated
METR says real-world ML research often has slower feedback and less clearly defined goals, leaving uncertainty about whether research can be packaged into small tasks for models to iterate on.
However, real-world ML R&D work often has much slower feedback loops and less clearly-defined goals than in our environments, and it’s unclear to what extent research problems can be packaged into small tasks that models can iterate on effectively.
Source-stated
METR presents RE-Bench as a possible tool for evaluations of dangerous levels of autonomous AI R&D capabilities.
We hope RE-Bench and the methodology we’ve developed will be useful in creating evaluations for dangerous levels of autonomous AI R&D capabilities.
Source and contributor context
METR · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
AI agent
An AI system that chooses and carries out actions toward a goal, often using software tools.
Machine learning
Methods that let computers learn patterns from examples instead of following rules written by hand.
METR
METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: 156ced09-5bc4-3d9d-3e04-00f1d44c1906.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-16T10:53:43.858Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model.
Observation: 81b12e818d59f4eeea6ab08acd215d5a54c85d83be5bfd727a232bc3b026b71a. Source text hash: e4c59bfca9d4ec2749a352d2bb84f65412b0c263b44a2997d62b59ce1d4aa742.
uncertain · 2026-11-16T10:53:43.858Z · anthropic/claude-opus-5.5
Publication: 156ced09-5bc4-3d9d-3e04-00f1d44c1906
attribution — pass: The reading is a single METR blog post announcing RE-Bench, and the reading attributes it to METR. Results and hopes are kept as METR's reported findings and statements.
We’re releasing RE-Bench, a new benchmark for measuring the performance of humans and frontier model agents on ML research engineering tasks.
summary — uncertain: The summary is faithful and preserves METR's main tension: agents win at 2 hours, humans win at longer budgets, and real-world transfer is uncertain. But 'scaffolding and orchestration' carries the cost-performance argument and is undefined. The source glosses scaffolds as setups that let a model repeatedly run code. Claude 3.5 Sonnet and o1-preview are also not identified as frontier models from Anthropic and OpenAI. Smallest repair: add a term-library entry for scaffolding and a short clause naming the models' developers.
Modular is a very simple baseline scaffolding that just lets the model repeatedly run code and see the results
date — pass: November 22, 2024 matches the UTC calendar day of evidence.originalPublished (2024-11-22T08:00Z) and the URL path. The kind is publisher-page, so no archive disclosure is needed. There is no hindsight framing. METR's expectation of better elicitation and its plans for more environments are reported as hopes or suggestions, not outcomes.
support — pass: Each claim is quoted closely and kept attributed: the short-budget agent advantage, the near-2x human lead at 32 hours, the cheaper hourly cost, the struggles with novel information, the real-world gaps and the dangerous-capability evaluation aim. The limitations correctly say the scaffolding gains are not established. The question follows directly from METR's own stated uncertainty.
it’s unclear to what extent research problems can be packaged into small tasks that models can iterate on effectively
media — pass: No media was selected, and no verified candidates were supplied. This is consistent with the decision record.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRAn AI system that chooses and carries out actions toward a goal, often using software tools.
Definition referenceMethods that let computers learn patterns from examples instead of following rules written by hand.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceMethods that let computers learn patterns from examples instead of following rules written by hand.
Definition referenceMethods that let computers learn patterns from examples instead of following rules written by hand.
Definition referenceMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRAn AI system that chooses and carries out actions toward a goal, often using software tools.
Definition referenceMethods that let computers learn patterns from examples instead of following rules written by hand.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceMethods that let computers learn patterns from examples instead of following rules written by hand.
Definition referenceMethods that let computers learn patterns from examples instead of following rules written by hand.
Definition reference