· Mar 15, 2024
METR shares a beta protocol for evaluating autonomous capabilities
Markdown · JSON · Original source
published this collection on March 15, 2024, to evaluate potentially dangerous autonomous capabilities of . It brings together example tasks, software tooling, guidance on capability elicitation, research on post-training enhancements, and a rough evaluation protocol. METR calls the protocol a “beta” and plans to improve it, while saying few concrete, executable proposals address these risks and frontier-model capabilities could increase substantially in the near future. The tasks focus on software engineering, engineering, cybersecurity, and research. METR estimates difficulty by how long a relevant human expert would take. Its guidance asks evaluators to try “elicitation,” described as post-training enhancements to achieve more capable agent performance, and account for failures further enhancements might fix. METR reports tentative results suggesting that post-training enhancements affect performance at a scale similar to the difference between GPT-3.5 Turbo and GPT-4. The collection is a starting point, but METR says the tasks are not closely tied to particular threat models and the suite has few tasks at the higher difficulty levels. It also warns that elicitation results may overstate or understate capabilities. How well the protocol’s scores correspond to capabilities relevant to concrete threat models remains unknown.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
· Article
Source details
Original source date: Mar 15, 2024 (2024-03-15T12:00:00.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-20T23:08:09.931Z. First observed: 2026-10-18T03:51:21.189Z. Retrieved: 2026-10-25T02:51:35.703Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
Text as retrieved on October 25, 2026; originally published March 15, 2024. says the task suite has only a small number of tasks, especially at the higher difficulty end, and that the tasks are not closely tied to particular threat models. METR says its results on the impact of post-training enhancements are tentative; the tasks used are not closely tied to a threat model beyond general autonomy, and their difficulty range may overstate the effect.
Open question
How well would the protocol’s scores identify capabilities relevant to specific threat models?
Source-stated
METR shares a collection of resources for evaluating potentially dangerous autonomous capabilities of frontier models.
METR is sharing a collection of resources for evaluating potentially dangerous autonomous capabilities of frontier models.
Source-stated
METR describes its example evaluation protocol as a beta that it plans to improve over time.
This protocol should be considered a “beta”, and we plan to iteratively improve it over time.
Source-stated
The example tasks cover software engineering, ML engineering, cybersecurity, and research, and range up to tasks estimated to take experienced humans a week.
The tasks mostly focus on areas where current frontier models are comparatively advantaged, and which are easy to set up and score: software engineering, ML engineering, cybersecurity, and research.
Source-stated
METR reports tentative results that post-training enhancements had an impact at a similar scale to the performance difference between GPT-3.5 Turbo and GPT-4.
We find that the impact of post-training enhancements is at a similar scale to the difference in performance between GPT-3.5 Turbo and GPT-4, although our results are tentative.
Source-stated
METR says the tasks are not closely tied to particular threat models.
The tasks are not that closely tied to particular threat models.
Source and contributor context
METR · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
Machine learning
Methods that let computers learn patterns from examples instead of following rules written by hand.
METR
METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Editorial provenance
Version 1. Publication ID: 7fb45555-0c70-00b7-a24a-bc4985c3d80a.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-20T23:04:15.927Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model, second_opinion_categorical.
Observation: f5c6860a78bc71b9b97be5675d5c877990218ae1424ee28498b8c8d65fec384e. Source text hash: d704efe06ad5ec93ccf35e9fd4ea0e14d4c2eb43a2c8c61fba013b4fed9c54d4.
Revision and source-check history
Version 1 · 2026-11-20T23:08:09.931Z
METR shares a beta protocol for evaluating autonomous capabilities
published this collection on March 15, 2024, to evaluate potentially dangerous autonomous capabilities of . It brings together example tasks, software tooling, guidance on capability elicitation, research on post-training enhancements, and a rough evaluation protocol. METR calls the protocol a “beta” and plans to improve it, while saying few concrete, executable proposals address these risks and frontier-model capabilities could increase substantially in the near future. The tasks focus on software engineering, engineering, cybersecurity, and research. METR estimates difficulty by how long a relevant human expert would take. Its guidance asks evaluators to try “elicitation,” described as post-training enhancements to achieve more capable agent performance, and account for failures further enhancements might fix. METR reports tentative results suggesting that post-training enhancements affect performance at a scale similar to the difference between GPT-3.5 Turbo and GPT-4. The collection is a starting point, but METR says the tasks are not closely tied to particular threat models and the suite has few tasks at the higher difficulty levels. It also warns that elicitation results may overstate or understate capabilities. How well the protocol’s scores correspond to capabilities relevant to concrete threat models remains unknown.
Original source
· Article
Source details
Original source date: Mar 15, 2024 (2024-03-15T12:00:00.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-20T23:08:09.931Z. First observed: 2026-10-18T03:51:21.189Z. Retrieved: 2026-10-25T02:51:35.703Z.
Limits
Text as retrieved on October 25, 2026; originally published March 15, 2024. says the task suite has only a small number of tasks, especially at the higher difficulty end, and that the tasks are not closely tied to particular threat models. METR says its results on the impact of post-training enhancements are tentative; the tasks used are not closely tied to a threat model beyond general autonomy, and their difficulty range may overstate the effect.
Open question
How well would the protocol’s scores identify capabilities relevant to specific threat models?
Source-stated
METR shares a collection of resources for evaluating potentially dangerous autonomous capabilities of frontier models.
METR is sharing a collection of resources for evaluating potentially dangerous autonomous capabilities of frontier models.
Source-stated
METR describes its example evaluation protocol as a beta that it plans to improve over time.
This protocol should be considered a “beta”, and we plan to iteratively improve it over time.
Source-stated
The example tasks cover software engineering, ML engineering, cybersecurity, and research, and range up to tasks estimated to take experienced humans a week.
The tasks mostly focus on areas where current frontier models are comparatively advantaged, and which are easy to set up and score: software engineering, ML engineering, cybersecurity, and research.
Source-stated
METR reports tentative results that post-training enhancements had an impact at a similar scale to the performance difference between GPT-3.5 Turbo and GPT-4.
We find that the impact of post-training enhancements is at a similar scale to the difference in performance between GPT-3.5 Turbo and GPT-4, although our results are tentative.
Source-stated
METR says the tasks are not closely tied to particular threat models.
The tasks are not that closely tied to particular threat models.
Source and contributor context
METR · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
Machine learning
Methods that let computers learn patterns from examples instead of following rules written by hand.
METR
METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Editorial provenance
Version 1. Publication ID: 7fb45555-0c70-00b7-a24a-bc4985c3d80a.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-20T23:04:15.927Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model, second_opinion_categorical.
Observation: f5c6860a78bc71b9b97be5675d5c877990218ae1424ee28498b8c8d65fec384e. Source text hash: d704efe06ad5ec93ccf35e9fd4ea0e14d4c2eb43a2c8c61fba013b4fed9c54d4.
uncertain · 2026-11-20T23:04:15.927Z · anthropic/claude-opus-5.5
Publication: 7fb45555-0c70-00b7-a24a-bc4985c3d80a
attribution — pass: This is a single METR blog post. The reading attributes the resources, the beta label, the task design, the tentative elicitation results and the stated limitations to METR, not to Escape Meridian or to established fact.
METR is sharing a collection of resources for evaluating potentially dangerous autonomous capabilities of frontier models.
summary — pass: The reading leads with METR and what it released, and keeps the beta framing and METR's stated urgency. It explains elicitation in METR's own words and keeps the tentative result and the limitations. Frontier model and ML are defined beside the reading. Threat model is left unexplained, but this does not block a general reader's understanding. The closing question follows from METR's own limitation.
This protocol should be considered a “beta”, and we plan to iteratively improve it over time.
date — uncertain: evidence.originalPublished is March 15, 2024, with basis backfill:feed_published. The source URL slug indicates 2024-03-13, two days earlier, which is more than a UTC timezone shift. The March 15 date may reflect a feed announcement rather than first publication, and the evidence text has no dateline to settle this. Otherwise there is no hindsight framing, forecasts are attributed to METR, and the page is not an archive copy.
support — pass: All claims match source text. The source does say both that tasks reach about a day of professional work and that they reach up to a week; the reading's one-week figure follows the strengths section, so this inconsistency is the source's. The possible overstatement or understatement, the small number of hard tasks, the weak link to threat models and the tentative results are all stated by METR. No benefits or costs are invented.
It may give an overestimate of model capabilities, if some failures are incorrectly treated as spurious.
media — pass: No media was selected, and no verified media candidates were supplied, so nothing needs checking.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METROne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceMethods that let computers learn patterns from examples instead of following rules written by hand.
Definition referenceMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METROne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceMethods that let computers learn patterns from examples instead of following rules written by hand.
Definition referenceMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METR