{"id":"a2f0d035f06f01504f78e0995d559a6f79d311b6eb533d854d9c755bec043dea","title":"METR shares a beta protocol for evaluating autonomous capabilities","argument":"METR published this collection on March 15, 2024, to evaluate potentially dangerous autonomous capabilities of frontier models. It brings together example tasks, software tooling, guidance on capability elicitation, research on post-training enhancements, and a rough evaluation protocol. METR calls the protocol a “beta” and plans to improve it, while saying few concrete, executable proposals address these risks and frontier-model capabilities could increase substantially in the near future.\n\nThe tasks focus on software engineering, ML engineering, cybersecurity, and research. METR estimates difficulty by how long a relevant human expert would take. Its guidance asks evaluators to try “elicitation,” described as post-training enhancements to achieve more capable agent performance, and account for failures further enhancements might fix.\n\nMETR reports tentative results suggesting that post-training enhancements affect performance at a scale similar to the difference between GPT-3.5 Turbo and GPT-4. The collection is a starting point, but METR says the tasks are not closely tied to particular threat models and the suite has few tasks at the higher difficulty levels. It also warns that elicitation results may overstate or understate capabilities. How well the protocol’s scores correspond to capabilities relevant to concrete threat models remains unknown.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 25, 2026; originally published March 15, 2024. METR says the task suite has only a small number of tasks, especially at the higher difficulty end, and that the tasks are not closely tied to particular threat models. METR says its results on the impact of post-training enhancements are tentative; the tasks used are not closely tied to a threat model beyond general autonomy, and their difficulty range may overstate the effect.","speaker":{"name":"METR","roleAtSource":"Publisher"},"source":{"title":"Autonomy Evaluation Resources","publisher":"METR","format":"Article","url":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/","publishedAt":"2024-03-15T12:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"METR shares a collection of resources for evaluating potentially dangerous autonomous capabilities of frontier models.","excerpt":"METR is sharing a collection of resources for evaluating potentially dangerous autonomous capabilities of frontier models.","url":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/","relationship":"Source-stated"},{"text":"METR describes its example evaluation protocol as a beta that it plans to improve over time.","excerpt":"This protocol should be considered a “beta”, and we plan to iteratively improve it over time.","url":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/","relationship":"Source-stated"},{"text":"The example tasks cover software engineering, ML engineering, cybersecurity, and research, and range up to tasks estimated to take experienced humans a week.","excerpt":"The tasks mostly focus on areas where current frontier models are comparatively advantaged, and which are easy to set up and score: software engineering, ML engineering, cybersecurity, and research.","url":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/","relationship":"Source-stated"},{"text":"METR reports tentative results that post-training enhancements had an impact at a similar scale to the performance difference between GPT-3.5 Turbo and GPT-4.","excerpt":"We find that the impact of post-training enhancements is at a similar scale to the difference in performance between GPT-3.5 Turbo and GPT-4, although our results are tentative.","url":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/","relationship":"Source-stated"},{"text":"METR says the tasks are not closely tied to particular threat models.","excerpt":"The tasks are not that closely tied to particular threat models.","url":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well would the protocol’s scores identify capabilities relevant to specific threat models?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"METR shares a beta protocol for evaluating autonomous capabilities","summary":"METR published this collection on March 15, 2024, to evaluate potentially dangerous autonomous capabilities of frontier models. It brings together example tasks, software tooling, guidance on capability elicitation, research on post-training enhancements, and a rough evaluation protocol. METR calls the protocol a “beta” and plans to improve it, while saying few concrete, executable proposals address these risks and frontier-model capabilities could increase substantially in the near future.\n\nThe tasks focus on software engineering, ML engineering, cybersecurity, and research. METR estimates difficulty by how long a relevant human expert would take. Its guidance asks evaluators to try “elicitation,” described as post-training enhancements to achieve more capable agent performance, and account for failures further enhancements might fix.\n\nMETR reports tentative results suggesting that post-training enhancements affect performance at a scale similar to the difference between GPT-3.5 Turbo and GPT-4. The collection is a starting point, but METR says the tasks are not closely tied to particular threat models and the suite has few tasks at the higher difficulty levels. It also warns that elicitation results may overstate or understate capabilities. How well the protocol’s scores correspond to capabilities relevant to concrete threat models remains unknown.","question":"How well would the protocol’s scores identify capabilities relevant to specific threat models?","claims":[{"text":"METR shares a collection of resources for evaluating potentially dangerous autonomous capabilities of frontier models.","excerpt":"METR is sharing a collection of resources for evaluating potentially dangerous autonomous capabilities of frontier models.","support":"source-stated"},{"text":"METR describes its example evaluation protocol as a beta that it plans to improve over time.","excerpt":"This protocol should be considered a “beta”, and we plan to iteratively improve it over time.","support":"source-stated"},{"text":"The example tasks cover software engineering, ML engineering, cybersecurity, and research, and range up to tasks estimated to take experienced humans a week.","excerpt":"The tasks mostly focus on areas where current frontier models are comparatively advantaged, and which are easy to set up and score: software engineering, ML engineering, cybersecurity, and research.","support":"source-stated"},{"text":"METR reports tentative results that post-training enhancements had an impact at a similar scale to the performance difference between GPT-3.5 Turbo and GPT-4.","excerpt":"We find that the impact of post-training enhancements is at a similar scale to the difference in performance between GPT-3.5 Turbo and GPT-4, although our results are tentative.","support":"source-stated"},{"text":"METR says the tasks are not closely tied to particular threat models.","excerpt":"The tasks are not that closely tied to particular threat models.","support":"source-stated"}],"limitations":["Text as retrieved on October 25, 2026; originally published March 15, 2024.","METR says the task suite has only a small number of tasks, especially at the higher difficulty end, and that the tasks are not closely tied to particular threat models.","METR says its results on the impact of post-training enhancements are tentative; the tasks used are not closely tied to a threat model beyond general autonomy, and their difficulty range may overstate the effect."],"mediaId":null,"correctionReason":null,"id":"a2f0d035f06f01504f78e0995d559a6f79d311b6eb533d854d9c755bec043dea","publicationId":"7fb45555-0c70-00b7-a24a-bc4985c3d80a","version":1,"sourceUrl":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/","sourceTitle":"Autonomy Evaluation Resources","publisher":"METR","publishedAt":"2024-03-15T12:00:00.000Z","originalPublished":{"value":"2024-03-15T12:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T03:51:21.189Z","createdAt":"2026-11-20T23:08:09.931Z","historical":true,"entityIds":["metr"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"c996d83d47cf519ec3d35f90f3ce9b4580908eb11cdefbd27a171f1fa090b505","rubricVersion":"unrecorded","provider":"unrecorded","model":"anthropic/claude-opus-5.5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-25T02:50:07.571Z"},"provenance":{"textHash":"d704efe06ad5ec93ccf35e9fd4ea0e14d4c2eb43a2c8c61fba013b4fed9c54d4","observationId":"f5c6860a78bc71b9b97be5675d5c877990218ae1424ee28498b8c8d65fec384e","retrievedAt":"2026-10-25T02:51:35.703Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"79fb0111-b7a7-e8a2-7e60-98c4f99936f1"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"cf851bf3-a220-a4f0-e85d-296885b046b9"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-20T23:04:15.927Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/metr-shares-a-beta-protocol-for-evaluating-autonomous-a2f0d035","formats":{"html":"/readings/metr-shares-a-beta-protocol-for-evaluating-autonomous-a2f0d035","markdown":"/readings/metr-shares-a-beta-protocol-for-evaluating-autonomous-a2f0d035.md","json":"/readings/metr-shares-a-beta-protocol-for-evaluating-autonomous-a2f0d035.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-frontier-model","name":"Frontier model","explanation":"One of the most capable AI models available at a given time; the set shifts as models improve.","referenceUrl":"https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper"},{"id":"term-machine-learning","name":"Machine learning","explanation":"Methods that let computers learn patterns from examples instead of following rules written by hand.","referenceUrl":"https://developers.google.com/machine-learning/glossary#machine-learning"}],"entities":[{"id":"metr","name":"METR","kind":"organization","context":"METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.","referenceUrls":["https://metr.org/"],"path":"/entities/metr","json":"/entities/metr.json","markdown":"/entities/metr.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"a2f0d035f06f01504f78e0995d559a6f79d311b6eb533d854d9c755bec043dea","revision":"7fb45555-0c70-00b7-a24a-bc4985c3d80a","revisionKind":"publication-id","title":"METR shares a beta protocol for evaluating autonomous capabilities","url":"https://escapemeridian.com/readings/metr-shares-a-beta-protocol-for-evaluating-autonomous-a2f0d035"},"source":{"type":"source","id":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/","revision":"d704efe06ad5ec93ccf35e9fd4ea0e14d4c2eb43a2c8c61fba013b4fed9c54d4","revisionKind":"sha256-source-text","title":"Autonomy Evaluation Resources","url":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"a2f0d035f06f01504f78e0995d559a6f79d311b6eb533d854d9c755bec043dea","title":"METR shares a beta protocol for evaluating autonomous capabilities","argument":"METR published this collection on March 15, 2024, to evaluate potentially dangerous autonomous capabilities of frontier models. It brings together example tasks, software tooling, guidance on capability elicitation, research on post-training enhancements, and a rough evaluation protocol. METR calls the protocol a “beta” and plans to improve it, while saying few concrete, executable proposals address these risks and frontier-model capabilities could increase substantially in the near future.\n\nThe tasks focus on software engineering, ML engineering, cybersecurity, and research. METR estimates difficulty by how long a relevant human expert would take. Its guidance asks evaluators to try “elicitation,” described as post-training enhancements to achieve more capable agent performance, and account for failures further enhancements might fix.\n\nMETR reports tentative results suggesting that post-training enhancements affect performance at a scale similar to the difference between GPT-3.5 Turbo and GPT-4. The collection is a starting point, but METR says the tasks are not closely tied to particular threat models and the suite has few tasks at the higher difficulty levels. It also warns that elicitation results may overstate or understate capabilities. How well the protocol’s scores correspond to capabilities relevant to concrete threat models remains unknown.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 25, 2026; originally published March 15, 2024. METR says the task suite has only a small number of tasks, especially at the higher difficulty end, and that the tasks are not closely tied to particular threat models. METR says its results on the impact of post-training enhancements are tentative; the tasks used are not closely tied to a threat model beyond general autonomy, and their difficulty range may overstate the effect.","speaker":{"name":"METR","roleAtSource":"Publisher"},"source":{"title":"Autonomy Evaluation Resources","publisher":"METR","format":"Article","url":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/","publishedAt":"2024-03-15T12:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"METR shares a collection of resources for evaluating potentially dangerous autonomous capabilities of frontier models.","excerpt":"METR is sharing a collection of resources for evaluating potentially dangerous autonomous capabilities of frontier models.","url":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/","relationship":"Source-stated"},{"text":"METR describes its example evaluation protocol as a beta that it plans to improve over time.","excerpt":"This protocol should be considered a “beta”, and we plan to iteratively improve it over time.","url":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/","relationship":"Source-stated"},{"text":"The example tasks cover software engineering, ML engineering, cybersecurity, and research, and range up to tasks estimated to take experienced humans a week.","excerpt":"The tasks mostly focus on areas where current frontier models are comparatively advantaged, and which are easy to set up and score: software engineering, ML engineering, cybersecurity, and research.","url":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/","relationship":"Source-stated"},{"text":"METR reports tentative results that post-training enhancements had an impact at a similar scale to the performance difference between GPT-3.5 Turbo and GPT-4.","excerpt":"We find that the impact of post-training enhancements is at a similar scale to the difference in performance between GPT-3.5 Turbo and GPT-4, although our results are tentative.","url":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/","relationship":"Source-stated"},{"text":"METR says the tasks are not closely tied to particular threat models.","excerpt":"The tasks are not that closely tied to particular threat models.","url":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well would the protocol’s scores identify capabilities relevant to specific threat models?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"METR shares a beta protocol for evaluating autonomous capabilities","summary":"METR published this collection on March 15, 2024, to evaluate potentially dangerous autonomous capabilities of frontier models. It brings together example tasks, software tooling, guidance on capability elicitation, research on post-training enhancements, and a rough evaluation protocol. METR calls the protocol a “beta” and plans to improve it, while saying few concrete, executable proposals address these risks and frontier-model capabilities could increase substantially in the near future.\n\nThe tasks focus on software engineering, ML engineering, cybersecurity, and research. METR estimates difficulty by how long a relevant human expert would take. Its guidance asks evaluators to try “elicitation,” described as post-training enhancements to achieve more capable agent performance, and account for failures further enhancements might fix.\n\nMETR reports tentative results suggesting that post-training enhancements affect performance at a scale similar to the difference between GPT-3.5 Turbo and GPT-4. The collection is a starting point, but METR says the tasks are not closely tied to particular threat models and the suite has few tasks at the higher difficulty levels. It also warns that elicitation results may overstate or understate capabilities. How well the protocol’s scores correspond to capabilities relevant to concrete threat models remains unknown.","question":"How well would the protocol’s scores identify capabilities relevant to specific threat models?","claims":[{"text":"METR shares a collection of resources for evaluating potentially dangerous autonomous capabilities of frontier models.","excerpt":"METR is sharing a collection of resources for evaluating potentially dangerous autonomous capabilities of frontier models.","support":"source-stated"},{"text":"METR describes its example evaluation protocol as a beta that it plans to improve over time.","excerpt":"This protocol should be considered a “beta”, and we plan to iteratively improve it over time.","support":"source-stated"},{"text":"The example tasks cover software engineering, ML engineering, cybersecurity, and research, and range up to tasks estimated to take experienced humans a week.","excerpt":"The tasks mostly focus on areas where current frontier models are comparatively advantaged, and which are easy to set up and score: software engineering, ML engineering, cybersecurity, and research.","support":"source-stated"},{"text":"METR reports tentative results that post-training enhancements had an impact at a similar scale to the performance difference between GPT-3.5 Turbo and GPT-4.","excerpt":"We find that the impact of post-training enhancements is at a similar scale to the difference in performance between GPT-3.5 Turbo and GPT-4, although our results are tentative.","support":"source-stated"},{"text":"METR says the tasks are not closely tied to particular threat models.","excerpt":"The tasks are not that closely tied to particular threat models.","support":"source-stated"}],"limitations":["Text as retrieved on October 25, 2026; originally published March 15, 2024.","METR says the task suite has only a small number of tasks, especially at the higher difficulty end, and that the tasks are not closely tied to particular threat models.","METR says its results on the impact of post-training enhancements are tentative; the tasks used are not closely tied to a threat model beyond general autonomy, and their difficulty range may overstate the effect."],"mediaId":null,"correctionReason":null,"id":"a2f0d035f06f01504f78e0995d559a6f79d311b6eb533d854d9c755bec043dea","publicationId":"7fb45555-0c70-00b7-a24a-bc4985c3d80a","version":1,"sourceUrl":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/","sourceTitle":"Autonomy Evaluation Resources","publisher":"METR","publishedAt":"2024-03-15T12:00:00.000Z","originalPublished":{"value":"2024-03-15T12:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T03:51:21.189Z","createdAt":"2026-11-20T23:08:09.931Z","historical":true,"entityIds":["metr"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2024-03-13-autonomy-evaluation-resources/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"c996d83d47cf519ec3d35f90f3ce9b4580908eb11cdefbd27a171f1fa090b505","rubricVersion":"unrecorded","provider":"unrecorded","model":"anthropic/claude-opus-5.5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-25T02:50:07.571Z"},"provenance":{"textHash":"d704efe06ad5ec93ccf35e9fd4ea0e14d4c2eb43a2c8c61fba013b4fed9c54d4","observationId":"f5c6860a78bc71b9b97be5675d5c877990218ae1424ee28498b8c8d65fec384e","retrievedAt":"2026-10-25T02:51:35.703Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"79fb0111-b7a7-e8a2-7e60-98c4f99936f1"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"cf851bf3-a220-a4f0-e85d-296885b046b9"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-20T23:04:15.927Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"7fb45555-0c70-00b7-a24a-bc4985c3d80a","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"This is a single METR blog post. The reading attributes the resources, the beta label, the task design, the tentative elicitation results and the stated limitations to METR, not to Escape Meridian or to established fact.","excerpt":"METR is sharing a collection of resources for evaluating potentially dangerous autonomous capabilities of frontier models."},{"dimension":"summary","status":"pass","explanation":"The reading leads with METR and what it released, and keeps the beta framing and METR's stated urgency. It explains elicitation in METR's own words and keeps the tentative result and the limitations. Frontier model and ML are defined beside the reading. Threat model is left unexplained, but this does not block a general reader's understanding. The closing question follows from METR's own limitation.","excerpt":"This protocol should be considered a “beta”, and we plan to iteratively improve it over time."},{"dimension":"date","status":"uncertain","explanation":"evidence.originalPublished is March 15, 2024, with basis backfill:feed_published. The source URL slug indicates 2024-03-13, two days earlier, which is more than a UTC timezone shift. The March 15 date may reflect a feed announcement rather than first publication, and the evidence text has no dateline to settle this. Otherwise there is no hindsight framing, forecasts are attributed to METR, and the page is not an archive copy.","excerpt":null},{"dimension":"support","status":"pass","explanation":"All claims match source text. The source does say both that tasks reach about a day of professional work and that they reach up to a week; the reading's one-week figure follows the strengths section, so this inconsistency is the source's. The possible overstatement or understatement, the small number of hard tasks, the weak link to threat models and the tentative results are all stated by METR. No benefits or costs are invented.","excerpt":"It may give an overestimate of model capabilities, if some failures are incorrectly treated as spurious."},{"dimension":"media","status":"pass","explanation":"No media was selected, and no verified media candidates were supplied, so nothing needs checking.","excerpt":null}],"createdAt":"2026-11-20T23:04:15.927Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}