{"id":"b8dc714c7a852a829da6b45a98d8cb16ce87adc6f7ddb2c591c7ea42a9023850","title":"METR reports on DeepSeek-V3’s autonomous capabilities","argument":"METR published this evaluation on February 12, 2025. It reports that DeepSeek-V3 performed on par with some earlier frontier models in tests of autonomous capabilities, and below others. METR found the model could handle programming and questions but struggled with agency and was sensitive to the scaffolding used to help it act. On RE-Bench, which tests AI R&D tasks, METR found its performance lower than GPT-4o, Claude 3, Claude 3.5 Sonnet, and o1. METR therefore judges DeepSeek-V3’s ability to perform AI R&D autonomously likely insufficient to significantly accelerate algorithmic AI progress. That conclusion is about these evaluations, not a demonstration of real-world performance. METR says it performed only basic elicitation and cannot confidently say whether the results represent the model’s full capabilities. It also reports that the model’s performance on GPQA was not the result of memorizing public questions. METR is not affiliated with DeepSeek and tested a version hosted by a third party.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 24, 2026; originally published February 12, 2025. METR says it used basic elicitation and cannot confidently say whether the evaluation represents the full extent of the model’s capabilities. METR conducted its tests on a model hosted by a third party and says it has no affiliation with DeepSeek.","speaker":{"name":"METR","roleAtSource":"Publisher"},"source":{"title":"DeepSeek-V3 Evaluation Results","publisher":"METR","format":"Article","url":"https://metr.org/evaluations/deepseek-v3-report/","publishedAt":"2025-02-12T08:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"METR says it found no significant evidence that DeepSeek-V3 has autonomous capabilities beyond those of existing models.","excerpt":"testing it for dual-use autonomous capabilities and failed to find significant evidence for a level of autonomous capabilities beyond those of existing models.","url":"https://metr.org/evaluations/deepseek-v3-report/","relationship":"Source-stated"},{"text":"METR reports that DeepSeek-V3’s AI R&D performance was lower than that of the listed models.","excerpt":"DeepSeek-V3’s performance on RE-Bench is lower than that of GPT-4o, Claude 3, Claude 3.5 Sonnet, and o1.","url":"https://metr.org/evaluations/deepseek-v3-report/","relationship":"Source-stated"},{"text":"METR says DeepSeek-V3’s autonomous AI R&D abilities are likely insufficient to significantly accelerate algorithmic AI progress.","excerpt":"we think DeepSeek-V3’s abilities to autonomously perform AI R&D are likely insufficient to significantly accelerate the rate of algorithmic AI progress.","url":"https://metr.org/evaluations/deepseek-v3-report/","relationship":"Source-stated"},{"text":"METR says its evaluation may not capture the full extent of DeepSeek-V3’s capabilities because it used only basic elicitation.","excerpt":"We performed only basic elicitation on DeepSeek-V3. Given this, it’s difficult to say with confidence whether the results here represent the full extent of the model’s capabilities.","url":"https://metr.org/evaluations/deepseek-v3-report/","relationship":"Source-stated"},{"text":"METR says its GPQA results suggest DeepSeek-V3’s performance was not due to memorizing public questions.","excerpt":"This suggests that DeepSeek-V3’s performance on GPQA (and the performance of other frontier models as well) is not the result of memorizing public questions.","url":"https://metr.org/evaluations/deepseek-v3-report/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/evaluations/deepseek-v3-report/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How much could further elicitation, including different scaffolding or post-training for agency, change DeepSeek-V3’s performance on these tasks?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"METR reports on DeepSeek-V3’s autonomous capabilities","summary":"METR published this evaluation on February 12, 2025. It reports that DeepSeek-V3 performed on par with some earlier frontier models in tests of autonomous capabilities, and below others. METR found the model could handle programming and questions but struggled with agency and was sensitive to the scaffolding used to help it act. On RE-Bench, which tests AI R&D tasks, METR found its performance lower than GPT-4o, Claude 3, Claude 3.5 Sonnet, and o1. METR therefore judges DeepSeek-V3’s ability to perform AI R&D autonomously likely insufficient to significantly accelerate algorithmic AI progress. That conclusion is about these evaluations, not a demonstration of real-world performance. METR says it performed only basic elicitation and cannot confidently say whether the results represent the model’s full capabilities. It also reports that the model’s performance on GPQA was not the result of memorizing public questions. METR is not affiliated with DeepSeek and tested a version hosted by a third party.","question":"How much could further elicitation, including different scaffolding or post-training for agency, change DeepSeek-V3’s performance on these tasks?","claims":[{"text":"METR says it found no significant evidence that DeepSeek-V3 has autonomous capabilities beyond those of existing models.","excerpt":"testing it for dual-use autonomous capabilities and failed to find significant evidence for a level of autonomous capabilities beyond those of existing models.","support":"source-stated"},{"text":"METR reports that DeepSeek-V3’s AI R&D performance was lower than that of the listed models.","excerpt":"DeepSeek-V3’s performance on RE-Bench is lower than that of GPT-4o, Claude 3, Claude 3.5 Sonnet, and o1.","support":"source-stated"},{"text":"METR says DeepSeek-V3’s autonomous AI R&D abilities are likely insufficient to significantly accelerate algorithmic AI progress.","excerpt":"we think DeepSeek-V3’s abilities to autonomously perform AI R&D are likely insufficient to significantly accelerate the rate of algorithmic AI progress.","support":"source-stated"},{"text":"METR says its evaluation may not capture the full extent of DeepSeek-V3’s capabilities because it used only basic elicitation.","excerpt":"We performed only basic elicitation on DeepSeek-V3. Given this, it’s difficult to say with confidence whether the results here represent the full extent of the model’s capabilities.","support":"source-stated"},{"text":"METR says its GPQA results suggest DeepSeek-V3’s performance was not due to memorizing public questions.","excerpt":"This suggests that DeepSeek-V3’s performance on GPQA (and the performance of other frontier models as well) is not the result of memorizing public questions.","support":"source-stated"}],"limitations":["Text as retrieved on October 24, 2026; originally published February 12, 2025.","METR says it used basic elicitation and cannot confidently say whether the evaluation represents the full extent of the model’s capabilities.","METR conducted its tests on a model hosted by a third party and says it has no affiliation with DeepSeek."],"mediaId":null,"correctionReason":null,"id":"b8dc714c7a852a829da6b45a98d8cb16ce87adc6f7ddb2c591c7ea42a9023850","publicationId":"e149b634-edc3-fc72-5cb5-22ecfc764497","version":1,"sourceUrl":"https://metr.org/evaluations/deepseek-v3-report/","sourceTitle":"DeepSeek-V3 Evaluation Results","publisher":"METR","publishedAt":"2025-02-12T08:00:00.000Z","originalPublished":{"value":"2025-02-12T08:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T03:39:22.959Z","createdAt":"2026-11-15T11:13:26.297Z","historical":true,"entityIds":["metr"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/evaluations/deepseek-v3-report/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"4a4328cbea60542a97794ab3cca4a185bd5373a846cf2d818b2d0ebf3f73324f","rubricVersion":"unrecorded","provider":"vercel","model":"typesafe-ai/jev","reasonCodes":["bridge_admitted_by_model"],"confidence":0.68,"decidedAt":"2026-10-24T01:01:35.481Z"},"provenance":{"textHash":"5460f4d1a5ce5777d1a348668da5be2f6f0f4a0494825df3c179f9c3ec14fcb1","observationId":"b75ae1f0a9a0cb5db8ec78cd321e7a4b8fb38a7c4433e596835195785bfd29e9","retrievedAt":"2026-10-24T01:03:39.430Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"e4c33a4d-f555-199a-cfa0-7280e010a543"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"124eb1d2-0f70-bfd4-67bf-5afb6a8e9888"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-15T11:08:23.859Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/metr-reports-on-deepseek-v3-s-autonomous-capabilities-b8dc714c","formats":{"html":"/readings/metr-reports-on-deepseek-v3-s-autonomous-capabilities-b8dc714c","markdown":"/readings/metr-reports-on-deepseek-v3-s-autonomous-capabilities-b8dc714c.md","json":"/readings/metr-reports-on-deepseek-v3-s-autonomous-capabilities-b8dc714c.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-frontier-model","name":"Frontier model","explanation":"One of the most capable AI models available at a given time; the set shifts as models improve.","referenceUrl":"https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper"}],"entities":[{"id":"metr","name":"METR","kind":"organization","context":"METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.","referenceUrls":["https://metr.org/"],"path":"/entities/metr","json":"/entities/metr.json","markdown":"/entities/metr.md"},{"id":"claude","name":"Claude","kind":"model","context":"Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.","referenceUrls":["https://claude.com/product/overview","https://www.anthropic.com/news","https://www.anthropic.com/research/team/economics"],"path":"/entities/claude","json":"/entities/claude.json","markdown":"/entities/claude.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"b8dc714c7a852a829da6b45a98d8cb16ce87adc6f7ddb2c591c7ea42a9023850","revision":"e149b634-edc3-fc72-5cb5-22ecfc764497","revisionKind":"publication-id","title":"METR reports on DeepSeek-V3’s autonomous capabilities","url":"https://escapemeridian.com/readings/metr-reports-on-deepseek-v3-s-autonomous-capabilities-b8dc714c"},"source":{"type":"source","id":"https://metr.org/evaluations/deepseek-v3-report/","revision":"5460f4d1a5ce5777d1a348668da5be2f6f0f4a0494825df3c179f9c3ec14fcb1","revisionKind":"sha256-source-text","title":"DeepSeek-V3 Evaluation Results","url":"https://metr.org/evaluations/deepseek-v3-report/"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"b8dc714c7a852a829da6b45a98d8cb16ce87adc6f7ddb2c591c7ea42a9023850","title":"METR reports on DeepSeek-V3’s autonomous capabilities","argument":"METR published this evaluation on February 12, 2025. It reports that DeepSeek-V3 performed on par with some earlier frontier models in tests of autonomous capabilities, and below others. METR found the model could handle programming and questions but struggled with agency and was sensitive to the scaffolding used to help it act. On RE-Bench, which tests AI R&D tasks, METR found its performance lower than GPT-4o, Claude 3, Claude 3.5 Sonnet, and o1. METR therefore judges DeepSeek-V3’s ability to perform AI R&D autonomously likely insufficient to significantly accelerate algorithmic AI progress. That conclusion is about these evaluations, not a demonstration of real-world performance. METR says it performed only basic elicitation and cannot confidently say whether the results represent the model’s full capabilities. It also reports that the model’s performance on GPQA was not the result of memorizing public questions. METR is not affiliated with DeepSeek and tested a version hosted by a third party.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 24, 2026; originally published February 12, 2025. METR says it used basic elicitation and cannot confidently say whether the evaluation represents the full extent of the model’s capabilities. METR conducted its tests on a model hosted by a third party and says it has no affiliation with DeepSeek.","speaker":{"name":"METR","roleAtSource":"Publisher"},"source":{"title":"DeepSeek-V3 Evaluation Results","publisher":"METR","format":"Article","url":"https://metr.org/evaluations/deepseek-v3-report/","publishedAt":"2025-02-12T08:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"METR says it found no significant evidence that DeepSeek-V3 has autonomous capabilities beyond those of existing models.","excerpt":"testing it for dual-use autonomous capabilities and failed to find significant evidence for a level of autonomous capabilities beyond those of existing models.","url":"https://metr.org/evaluations/deepseek-v3-report/","relationship":"Source-stated"},{"text":"METR reports that DeepSeek-V3’s AI R&D performance was lower than that of the listed models.","excerpt":"DeepSeek-V3’s performance on RE-Bench is lower than that of GPT-4o, Claude 3, Claude 3.5 Sonnet, and o1.","url":"https://metr.org/evaluations/deepseek-v3-report/","relationship":"Source-stated"},{"text":"METR says DeepSeek-V3’s autonomous AI R&D abilities are likely insufficient to significantly accelerate algorithmic AI progress.","excerpt":"we think DeepSeek-V3’s abilities to autonomously perform AI R&D are likely insufficient to significantly accelerate the rate of algorithmic AI progress.","url":"https://metr.org/evaluations/deepseek-v3-report/","relationship":"Source-stated"},{"text":"METR says its evaluation may not capture the full extent of DeepSeek-V3’s capabilities because it used only basic elicitation.","excerpt":"We performed only basic elicitation on DeepSeek-V3. Given this, it’s difficult to say with confidence whether the results here represent the full extent of the model’s capabilities.","url":"https://metr.org/evaluations/deepseek-v3-report/","relationship":"Source-stated"},{"text":"METR says its GPQA results suggest DeepSeek-V3’s performance was not due to memorizing public questions.","excerpt":"This suggests that DeepSeek-V3’s performance on GPQA (and the performance of other frontier models as well) is not the result of memorizing public questions.","url":"https://metr.org/evaluations/deepseek-v3-report/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/evaluations/deepseek-v3-report/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How much could further elicitation, including different scaffolding or post-training for agency, change DeepSeek-V3’s performance on these tasks?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"METR reports on DeepSeek-V3’s autonomous capabilities","summary":"METR published this evaluation on February 12, 2025. It reports that DeepSeek-V3 performed on par with some earlier frontier models in tests of autonomous capabilities, and below others. METR found the model could handle programming and questions but struggled with agency and was sensitive to the scaffolding used to help it act. On RE-Bench, which tests AI R&D tasks, METR found its performance lower than GPT-4o, Claude 3, Claude 3.5 Sonnet, and o1. METR therefore judges DeepSeek-V3’s ability to perform AI R&D autonomously likely insufficient to significantly accelerate algorithmic AI progress. That conclusion is about these evaluations, not a demonstration of real-world performance. METR says it performed only basic elicitation and cannot confidently say whether the results represent the model’s full capabilities. It also reports that the model’s performance on GPQA was not the result of memorizing public questions. METR is not affiliated with DeepSeek and tested a version hosted by a third party.","question":"How much could further elicitation, including different scaffolding or post-training for agency, change DeepSeek-V3’s performance on these tasks?","claims":[{"text":"METR says it found no significant evidence that DeepSeek-V3 has autonomous capabilities beyond those of existing models.","excerpt":"testing it for dual-use autonomous capabilities and failed to find significant evidence for a level of autonomous capabilities beyond those of existing models.","support":"source-stated"},{"text":"METR reports that DeepSeek-V3’s AI R&D performance was lower than that of the listed models.","excerpt":"DeepSeek-V3’s performance on RE-Bench is lower than that of GPT-4o, Claude 3, Claude 3.5 Sonnet, and o1.","support":"source-stated"},{"text":"METR says DeepSeek-V3’s autonomous AI R&D abilities are likely insufficient to significantly accelerate algorithmic AI progress.","excerpt":"we think DeepSeek-V3’s abilities to autonomously perform AI R&D are likely insufficient to significantly accelerate the rate of algorithmic AI progress.","support":"source-stated"},{"text":"METR says its evaluation may not capture the full extent of DeepSeek-V3’s capabilities because it used only basic elicitation.","excerpt":"We performed only basic elicitation on DeepSeek-V3. Given this, it’s difficult to say with confidence whether the results here represent the full extent of the model’s capabilities.","support":"source-stated"},{"text":"METR says its GPQA results suggest DeepSeek-V3’s performance was not due to memorizing public questions.","excerpt":"This suggests that DeepSeek-V3’s performance on GPQA (and the performance of other frontier models as well) is not the result of memorizing public questions.","support":"source-stated"}],"limitations":["Text as retrieved on October 24, 2026; originally published February 12, 2025.","METR says it used basic elicitation and cannot confidently say whether the evaluation represents the full extent of the model’s capabilities.","METR conducted its tests on a model hosted by a third party and says it has no affiliation with DeepSeek."],"mediaId":null,"correctionReason":null,"id":"b8dc714c7a852a829da6b45a98d8cb16ce87adc6f7ddb2c591c7ea42a9023850","publicationId":"e149b634-edc3-fc72-5cb5-22ecfc764497","version":1,"sourceUrl":"https://metr.org/evaluations/deepseek-v3-report/","sourceTitle":"DeepSeek-V3 Evaluation Results","publisher":"METR","publishedAt":"2025-02-12T08:00:00.000Z","originalPublished":{"value":"2025-02-12T08:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T03:39:22.959Z","createdAt":"2026-11-15T11:13:26.297Z","historical":true,"entityIds":["metr"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/evaluations/deepseek-v3-report/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"4a4328cbea60542a97794ab3cca4a185bd5373a846cf2d818b2d0ebf3f73324f","rubricVersion":"unrecorded","provider":"vercel","model":"typesafe-ai/jev","reasonCodes":["bridge_admitted_by_model"],"confidence":0.68,"decidedAt":"2026-10-24T01:01:35.481Z"},"provenance":{"textHash":"5460f4d1a5ce5777d1a348668da5be2f6f0f4a0494825df3c179f9c3ec14fcb1","observationId":"b75ae1f0a9a0cb5db8ec78cd321e7a4b8fb38a7c4433e596835195785bfd29e9","retrievedAt":"2026-10-24T01:03:39.430Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"e4c33a4d-f555-199a-cfa0-7280e010a543"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"124eb1d2-0f70-bfd4-67bf-5afb6a8e9888"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-15T11:08:23.859Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"e149b634-edc3-fc72-5cb5-22ecfc764497","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"This is a single METR evaluation report on DeepSeek-V3. The reading attributes the findings, conclusions and caveats to METR throughout, and correctly notes METR's lack of affiliation with DeepSeek and its use of a third-party host.","excerpt":"METR has no affiliation with DeepSeek and conducted our tests on a model hosted by a 3rd party."},{"dimension":"summary","status":"uncertain","explanation":"Two central terms go unexplained. 'Elicitation' anchors both the caveat and the question; the source defines it as modifying an agent's scaffolding to give it the best chance of succeeding. 'GPQA' appears with no indication that it is a question benchmark, or why memorization would matter. The reading also drops METR's main framing point: open-weight models now match state-of-the-art models from 6 months earlier, after previously lagging much further. Smallest repair: add one-clause glosses for elicitation and GPQA, and one sentence on METR's open-weight gap observation.","excerpt":"Overall, the performance of open-weight models on our tasks is on par with state-of-the-art models from 6 months ago, whereas they had previously lagged significantly further behind."},{"dimension":"date","status":"pass","explanation":"originalPublished is 2025-02-12T08:00Z, and the reading states February 12, 2025, which matches the UTC day. The evidence kind is publisher-page, so no archive disclosure is needed. The retrieval date is given only as a capture note. There is no hindsight framing, and forecasts are not presented as outcomes.","excerpt":null},{"dimension":"support","status":"uncertain","explanation":"The claim that RE-Bench performance was lower than GPT-4o and Claude 3 quotes METR's Results section verbatim. However, METR's own summary says DeepSeek-V3 was on par with Claude 3 Opus and very slightly better than GPT-4o. The source is internally inconsistent, and the reading presents only one side. Repair: note the conflict, or say it was comparable to Claude 3 Opus and well below Claude 3.5 Sonnet and o1. The other claims match their excerpts.","excerpt":"DeepSeek-V3’s performance on AI R&D tasks is on-par with the performance of Claude 3 Opus, very slightly better than GPT-4o, and substantially worse than"},{"dimension":"media","status":"pass","explanation":"No media was selected and no verified candidates were supplied. Omitting media is appropriate.","excerpt":null}],"createdAt":"2026-11-15T11:08:23.859Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}