{"id":"e4b827bc267c64367e439179df0491d06f5c7ede1336b9e617a3482640ffed67","title":"METR reports DeepSeek-R1’s results on autonomous capabilities","argument":"METR published this evaluation on March 5, 2025. It reports that DeepSeek-R1 performed slightly better than o1-preview on its general autonomy tasks, but that it found no significant evidence of autonomous capabilities beyond existing models. On RE-Bench, DeepSeek-R1 performed lower than GPT-4o, Claude 3, Claude 3.5 Sonnet and o1. METR concludes that the model’s abilities to autonomously perform AI R&D are likely insufficient to significantly accelerate the rate of algorithmic AI progress. It also describes recurring problems with instruction following and recovering from mistakes. These results concern METR’s task suites: the report says its tasks may be too artificial to capture real-world demands, and that moderate elicitation makes it difficult to know whether the results represent the model’s full capabilities. METR says further elicitation might improve performance, but how much it would change the evaluation results remains unknown.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 24, 2026; originally published March 5, 2025. METR says it performed only moderate elicitation and that it is difficult to know whether the results represent the full extent of the model’s capabilities.","speaker":{"name":"METR","roleAtSource":"Publisher"},"source":{"title":"DeepSeek-R1 Evaluation Results","publisher":"METR","format":"Article","url":"https://metr.org/evaluations/deepseek-r1-report/","publishedAt":"2025-03-05T08:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"METR says DeepSeek-R1’s autonomous performance did not show strong evidence of capabilities beyond existing models.","excerpt":"We failed to find significant evidence for a level of autonomous capabilities beyond those of existing models.","url":"https://metr.org/evaluations/deepseek-r1-report/","relationship":"Source-stated"},{"text":"METR says DeepSeek-R1’s AI R&D abilities are likely insufficient to significantly accelerate algorithmic AI progress.","excerpt":"we think DeepSeek-R1’s abilities to autonomously perform AI R&D are likely insufficient to significantly accelerate the rate of algorithmic AI progress.","url":"https://metr.org/evaluations/deepseek-r1-report/","relationship":"Source-stated"},{"text":"METR reports recurring problems with instruction following and recovering from mistakes.","excerpt":"DeepSeek-R1 is not good at recovering from mistakes it makes in its code or the course of action it chooses to take.","url":"https://metr.org/evaluations/deepseek-r1-report/","relationship":"Source-stated"},{"text":"METR says further elicitation might unlock significant capabilities.","excerpt":"Further elicitation might unlock significant capabilities: DeepSeek-R1is sensitive to elicitation as we were able to noticeably improve the agent’s performance on the dev set over less than 1 week of elicitation.","url":"https://metr.org/evaluations/deepseek-r1-report/","relationship":"Source-stated"},{"text":"METR says its tasks may not capture the demands of real-world autonomous replication and adaptation.","excerpt":"Our tasks may be too artificial to capture the nuances of what’s required for real-world autonomous replication and adaptation","url":"https://metr.org/evaluations/deepseek-r1-report/","relationship":"Source-stated"},{"text":"METR reports that DeepSeek-R1 performed lower on RE-Bench than GPT-4o, Claude 3, Claude 3.5 Sonnet, and o1.","excerpt":"DeepSeek-R1’s performance on RE-Bench is lower than that of GPT-4o, Claude 3, Claude 3.5 Sonnet, and o1.","url":"https://metr.org/evaluations/deepseek-r1-report/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/evaluations/deepseek-r1-report/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How much would further elicitation change DeepSeek-R1’s performance on these evaluations?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"METR reports DeepSeek-R1’s results on autonomous capabilities","summary":"METR published this evaluation on March 5, 2025. It reports that DeepSeek-R1 performed slightly better than o1-preview on its general autonomy tasks, but that it found no significant evidence of autonomous capabilities beyond existing models. On RE-Bench, DeepSeek-R1 performed lower than GPT-4o, Claude 3, Claude 3.5 Sonnet and o1. METR concludes that the model’s abilities to autonomously perform AI R&D are likely insufficient to significantly accelerate the rate of algorithmic AI progress. It also describes recurring problems with instruction following and recovering from mistakes. These results concern METR’s task suites: the report says its tasks may be too artificial to capture real-world demands, and that moderate elicitation makes it difficult to know whether the results represent the model’s full capabilities. METR says further elicitation might improve performance, but how much it would change the evaluation results remains unknown.","question":"How much would further elicitation change DeepSeek-R1’s performance on these evaluations?","claims":[{"text":"METR says DeepSeek-R1’s autonomous performance did not show strong evidence of capabilities beyond existing models.","excerpt":"We failed to find significant evidence for a level of autonomous capabilities beyond those of existing models.","support":"source-stated"},{"text":"METR says DeepSeek-R1’s AI R&D abilities are likely insufficient to significantly accelerate algorithmic AI progress.","excerpt":"we think DeepSeek-R1’s abilities to autonomously perform AI R&D are likely insufficient to significantly accelerate the rate of algorithmic AI progress.","support":"source-stated"},{"text":"METR reports recurring problems with instruction following and recovering from mistakes.","excerpt":"DeepSeek-R1 is not good at recovering from mistakes it makes in its code or the course of action it chooses to take.","support":"source-stated"},{"text":"METR says further elicitation might unlock significant capabilities.","excerpt":"Further elicitation might unlock significant capabilities: DeepSeek-R1is sensitive to elicitation as we were able to noticeably improve the agent’s performance on the dev set over less than 1 week of elicitation.","support":"source-stated"},{"text":"METR says its tasks may not capture the demands of real-world autonomous replication and adaptation.","excerpt":"Our tasks may be too artificial to capture the nuances of what’s required for real-world autonomous replication and adaptation","support":"source-stated"},{"text":"METR reports that DeepSeek-R1 performed lower on RE-Bench than GPT-4o, Claude 3, Claude 3.5 Sonnet, and o1.","excerpt":"DeepSeek-R1’s performance on RE-Bench is lower than that of GPT-4o, Claude 3, Claude 3.5 Sonnet, and o1.","support":"source-stated"}],"limitations":["Text as retrieved on October 24, 2026; originally published March 5, 2025.","METR says it performed only moderate elicitation and that it is difficult to know whether the results represent the full extent of the model’s capabilities."],"mediaId":null,"correctionReason":null,"id":"e4b827bc267c64367e439179df0491d06f5c7ede1336b9e617a3482640ffed67","publicationId":"5402502d-df3f-cb59-8c9a-1791490ce534","version":1,"sourceUrl":"https://metr.org/evaluations/deepseek-r1-report/","sourceTitle":"DeepSeek-R1 Evaluation Results","publisher":"METR","publishedAt":"2025-03-05T08:00:00.000Z","originalPublished":{"value":"2025-03-05T08:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T03:37:21.633Z","createdAt":"2026-11-15T02:28:23.654Z","historical":true,"entityIds":["metr"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/evaluations/deepseek-r1-report/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"de8bfd1b26b3ef76a41eb1f107df9a0ac6fd186f981d79acd2f56c037e658d93","rubricVersion":"unrecorded","provider":"vercel","model":"typesafe-ai/jev","reasonCodes":["bridge_admitted_by_model"],"confidence":0.58,"decidedAt":"2026-10-24T00:23:40.654Z"},"provenance":{"textHash":"e630bdc12b1167ba342eeb4f8d7571f64b65a53d29c528212d87bb6a87c649df","observationId":"bc1800cf9e36ee08ab81eebd3821916db3646905ec6589cc3bf50bafa05cfbfd","retrievedAt":"2026-10-24T00:25:35.785Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"fe7f8a7b-da91-4a4c-fec1-703f738401b1"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"cd8435f7-7bf4-278a-9151-1d9b9995ba89"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-15T02:23:41.946Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/metr-reports-deepseek-r1-s-results-on-autonomous-capabilities-e4b827bc","formats":{"html":"/readings/metr-reports-deepseek-r1-s-results-on-autonomous-capabilities-e4b827bc","markdown":"/readings/metr-reports-deepseek-r1-s-results-on-autonomous-capabilities-e4b827bc.md","json":"/readings/metr-reports-deepseek-r1-s-results-on-autonomous-capabilities-e4b827bc.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[],"entities":[{"id":"metr","name":"METR","kind":"organization","context":"METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.","referenceUrls":["https://metr.org/"],"path":"/entities/metr","json":"/entities/metr.json","markdown":"/entities/metr.md"},{"id":"claude","name":"Claude","kind":"model","context":"Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.","referenceUrls":["https://claude.com/product/overview","https://www.anthropic.com/news","https://www.anthropic.com/research/team/economics"],"path":"/entities/claude","json":"/entities/claude.json","markdown":"/entities/claude.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"e4b827bc267c64367e439179df0491d06f5c7ede1336b9e617a3482640ffed67","revision":"5402502d-df3f-cb59-8c9a-1791490ce534","revisionKind":"publication-id","title":"METR reports DeepSeek-R1’s results on autonomous capabilities","url":"https://escapemeridian.com/readings/metr-reports-deepseek-r1-s-results-on-autonomous-capabilities-e4b827bc"},"source":{"type":"source","id":"https://metr.org/evaluations/deepseek-r1-report/","revision":"e630bdc12b1167ba342eeb4f8d7571f64b65a53d29c528212d87bb6a87c649df","revisionKind":"sha256-source-text","title":"DeepSeek-R1 Evaluation Results","url":"https://metr.org/evaluations/deepseek-r1-report/"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"e4b827bc267c64367e439179df0491d06f5c7ede1336b9e617a3482640ffed67","title":"METR reports DeepSeek-R1’s results on autonomous capabilities","argument":"METR published this evaluation on March 5, 2025. It reports that DeepSeek-R1 performed slightly better than o1-preview on its general autonomy tasks, but that it found no significant evidence of autonomous capabilities beyond existing models. On RE-Bench, DeepSeek-R1 performed lower than GPT-4o, Claude 3, Claude 3.5 Sonnet and o1. METR concludes that the model’s abilities to autonomously perform AI R&D are likely insufficient to significantly accelerate the rate of algorithmic AI progress. It also describes recurring problems with instruction following and recovering from mistakes. These results concern METR’s task suites: the report says its tasks may be too artificial to capture real-world demands, and that moderate elicitation makes it difficult to know whether the results represent the model’s full capabilities. METR says further elicitation might improve performance, but how much it would change the evaluation results remains unknown.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 24, 2026; originally published March 5, 2025. METR says it performed only moderate elicitation and that it is difficult to know whether the results represent the full extent of the model’s capabilities.","speaker":{"name":"METR","roleAtSource":"Publisher"},"source":{"title":"DeepSeek-R1 Evaluation Results","publisher":"METR","format":"Article","url":"https://metr.org/evaluations/deepseek-r1-report/","publishedAt":"2025-03-05T08:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"METR says DeepSeek-R1’s autonomous performance did not show strong evidence of capabilities beyond existing models.","excerpt":"We failed to find significant evidence for a level of autonomous capabilities beyond those of existing models.","url":"https://metr.org/evaluations/deepseek-r1-report/","relationship":"Source-stated"},{"text":"METR says DeepSeek-R1’s AI R&D abilities are likely insufficient to significantly accelerate algorithmic AI progress.","excerpt":"we think DeepSeek-R1’s abilities to autonomously perform AI R&D are likely insufficient to significantly accelerate the rate of algorithmic AI progress.","url":"https://metr.org/evaluations/deepseek-r1-report/","relationship":"Source-stated"},{"text":"METR reports recurring problems with instruction following and recovering from mistakes.","excerpt":"DeepSeek-R1 is not good at recovering from mistakes it makes in its code or the course of action it chooses to take.","url":"https://metr.org/evaluations/deepseek-r1-report/","relationship":"Source-stated"},{"text":"METR says further elicitation might unlock significant capabilities.","excerpt":"Further elicitation might unlock significant capabilities: DeepSeek-R1is sensitive to elicitation as we were able to noticeably improve the agent’s performance on the dev set over less than 1 week of elicitation.","url":"https://metr.org/evaluations/deepseek-r1-report/","relationship":"Source-stated"},{"text":"METR says its tasks may not capture the demands of real-world autonomous replication and adaptation.","excerpt":"Our tasks may be too artificial to capture the nuances of what’s required for real-world autonomous replication and adaptation","url":"https://metr.org/evaluations/deepseek-r1-report/","relationship":"Source-stated"},{"text":"METR reports that DeepSeek-R1 performed lower on RE-Bench than GPT-4o, Claude 3, Claude 3.5 Sonnet, and o1.","excerpt":"DeepSeek-R1’s performance on RE-Bench is lower than that of GPT-4o, Claude 3, Claude 3.5 Sonnet, and o1.","url":"https://metr.org/evaluations/deepseek-r1-report/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/evaluations/deepseek-r1-report/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How much would further elicitation change DeepSeek-R1’s performance on these evaluations?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"METR reports DeepSeek-R1’s results on autonomous capabilities","summary":"METR published this evaluation on March 5, 2025. It reports that DeepSeek-R1 performed slightly better than o1-preview on its general autonomy tasks, but that it found no significant evidence of autonomous capabilities beyond existing models. On RE-Bench, DeepSeek-R1 performed lower than GPT-4o, Claude 3, Claude 3.5 Sonnet and o1. METR concludes that the model’s abilities to autonomously perform AI R&D are likely insufficient to significantly accelerate the rate of algorithmic AI progress. It also describes recurring problems with instruction following and recovering from mistakes. These results concern METR’s task suites: the report says its tasks may be too artificial to capture real-world demands, and that moderate elicitation makes it difficult to know whether the results represent the model’s full capabilities. METR says further elicitation might improve performance, but how much it would change the evaluation results remains unknown.","question":"How much would further elicitation change DeepSeek-R1’s performance on these evaluations?","claims":[{"text":"METR says DeepSeek-R1’s autonomous performance did not show strong evidence of capabilities beyond existing models.","excerpt":"We failed to find significant evidence for a level of autonomous capabilities beyond those of existing models.","support":"source-stated"},{"text":"METR says DeepSeek-R1’s AI R&D abilities are likely insufficient to significantly accelerate algorithmic AI progress.","excerpt":"we think DeepSeek-R1’s abilities to autonomously perform AI R&D are likely insufficient to significantly accelerate the rate of algorithmic AI progress.","support":"source-stated"},{"text":"METR reports recurring problems with instruction following and recovering from mistakes.","excerpt":"DeepSeek-R1 is not good at recovering from mistakes it makes in its code or the course of action it chooses to take.","support":"source-stated"},{"text":"METR says further elicitation might unlock significant capabilities.","excerpt":"Further elicitation might unlock significant capabilities: DeepSeek-R1is sensitive to elicitation as we were able to noticeably improve the agent’s performance on the dev set over less than 1 week of elicitation.","support":"source-stated"},{"text":"METR says its tasks may not capture the demands of real-world autonomous replication and adaptation.","excerpt":"Our tasks may be too artificial to capture the nuances of what’s required for real-world autonomous replication and adaptation","support":"source-stated"},{"text":"METR reports that DeepSeek-R1 performed lower on RE-Bench than GPT-4o, Claude 3, Claude 3.5 Sonnet, and o1.","excerpt":"DeepSeek-R1’s performance on RE-Bench is lower than that of GPT-4o, Claude 3, Claude 3.5 Sonnet, and o1.","support":"source-stated"}],"limitations":["Text as retrieved on October 24, 2026; originally published March 5, 2025.","METR says it performed only moderate elicitation and that it is difficult to know whether the results represent the full extent of the model’s capabilities."],"mediaId":null,"correctionReason":null,"id":"e4b827bc267c64367e439179df0491d06f5c7ede1336b9e617a3482640ffed67","publicationId":"5402502d-df3f-cb59-8c9a-1791490ce534","version":1,"sourceUrl":"https://metr.org/evaluations/deepseek-r1-report/","sourceTitle":"DeepSeek-R1 Evaluation Results","publisher":"METR","publishedAt":"2025-03-05T08:00:00.000Z","originalPublished":{"value":"2025-03-05T08:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T03:37:21.633Z","createdAt":"2026-11-15T02:28:23.654Z","historical":true,"entityIds":["metr"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/evaluations/deepseek-r1-report/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"de8bfd1b26b3ef76a41eb1f107df9a0ac6fd186f981d79acd2f56c037e658d93","rubricVersion":"unrecorded","provider":"vercel","model":"typesafe-ai/jev","reasonCodes":["bridge_admitted_by_model"],"confidence":0.58,"decidedAt":"2026-10-24T00:23:40.654Z"},"provenance":{"textHash":"e630bdc12b1167ba342eeb4f8d7571f64b65a53d29c528212d87bb6a87c649df","observationId":"bc1800cf9e36ee08ab81eebd3821916db3646905ec6589cc3bf50bafa05cfbfd","retrievedAt":"2026-10-24T00:25:35.785Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"fe7f8a7b-da91-4a4c-fec1-703f738401b1"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"cd8435f7-7bf4-278a-9151-1d9b9995ba89"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-15T02:23:41.946Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"5402502d-df3f-cb59-8c9a-1791490ce534","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading consistently attributes the findings, conclusions and caveats to METR, the publisher of this single evaluation report. It keeps METR's assessments as METR's statements rather than established facts.","excerpt":"We performed two evaluations of DeepSeek-R1 testing for dual-use autonomous capabilities"},{"dimension":"summary","status":"uncertain","explanation":"Central terms are left unexplained, and no definitions are supplied. 'Elicitation' anchors both the closing caveat and the question, yet a general reader is not told it means adjusting the scaffolding around a model to give it its best chance on the tasks. 'RE-Bench', 'AI R&D', 'autonomous capabilities' and 'o1-preview' are also unexplained. Smallest repair: add short plain-language glosses or term-library tooltips. The reading also omits METR's headline significance: open-weight models are on par with state-of-the-art models from 7 months earlier but 5-28x cheaper. Restoring that clause would preserve why the report matters.","excerpt":"We call the process of modifying the scaffolding around an agent to give it the best shot at succeeding at our task suite “elicitation”"},{"dimension":"date","status":"pass","explanation":"The reading states March 5, 2025, which is the UTC calendar day of evidence.originalPublished, at day precision. Nothing later is presented as known, and there is no hindsight framing. METR's judgements about future elicitation are kept as METR's statements. The page is a publisher page, so no archive disclosure is needed.","excerpt":null},{"dimension":"support","status":"pass","explanation":"Each claim matches the source. This includes: no significant evidence of capabilities beyond existing models; lower RE-Bench performance than GPT-4o, Claude 3, Claude 3.5 Sonnet and o1; AI R&D abilities likely insufficient to accelerate progress; repeated failure modes in instruction following and recovery from mistakes; and the artificiality and elicitation caveats. The source's earlier line calls R1 comparable to GPT-4o on RE-Bench, while its results section says lower. That inconsistency is the source's, and the reading quotes the results section accurately. The question follows from METR's stated uncertainty.","excerpt":"In our experience eliciting DeepSeek-R1 and inspecting its outputs, there are a few common failure modes we noticed coming up repeatedly"},{"dimension":"media","status":"pass","explanation":"No media is selected, and no verified media candidates were supplied. Omitting media is appropriate.","excerpt":null}],"createdAt":"2026-11-15T02:23:41.946Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}