· Feb 12, 2025
METR reports on DeepSeek-V3’s autonomous capabilities
Markdown · JSON · Original source
published this evaluation on February 12, 2025. It reports that DeepSeek-V3 performed on par with some earlier in tests of autonomous capabilities, and below others. METR found the model could handle programming and questions but struggled with agency and was sensitive to the scaffolding used to help it act. On RE-Bench, which tests AI R&D tasks, METR found its performance lower than GPT-4o, 3, Claude 3.5 Sonnet, and o1. METR therefore judges DeepSeek-V3’s ability to perform AI R&D autonomously likely insufficient to significantly accelerate algorithmic AI progress. That conclusion is about these evaluations, not a demonstration of real-world performance. METR says it performed only basic elicitation and cannot confidently say whether the results represent the model’s full capabilities. It also reports that the model’s performance on GPQA was not the result of memorizing public questions. METR is not affiliated with DeepSeek and tested a version hosted by a third party.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
· Article
Source details
Original source date: Feb 12, 2025 (2025-02-12T08:00:00.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-15T11:13:26.297Z. First observed: 2026-10-18T03:39:22.959Z. Retrieved: 2026-10-24T01:03:39.430Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
Text as retrieved on October 24, 2026; originally published February 12, 2025. says it used basic elicitation and cannot confidently say whether the evaluation represents the full extent of the model’s capabilities. METR conducted its tests on a model hosted by a third party and says it has no affiliation with DeepSeek.
Open question
How much could further elicitation, including different scaffolding or post-training for agency, change DeepSeek-V3’s performance on these tasks?
Source-stated
METR says it found no significant evidence that DeepSeek-V3 has autonomous capabilities beyond those of existing models.
testing it for dual-use autonomous capabilities and failed to find significant evidence for a level of autonomous capabilities beyond those of existing models.
Source-stated
METR reports that DeepSeek-V3’s AI R&D performance was lower than that of the listed models.
DeepSeek-V3’s performance on RE-Bench is lower than that of GPT-4o, Claude 3, Claude 3.5 Sonnet, and o1.
Source-stated
METR says DeepSeek-V3’s autonomous AI R&D abilities are likely insufficient to significantly accelerate algorithmic AI progress.
we think DeepSeek-V3’s abilities to autonomously perform AI R&D are likely insufficient to significantly accelerate the rate of algorithmic AI progress.
Source-stated
METR says its evaluation may not capture the full extent of DeepSeek-V3’s capabilities because it used only basic elicitation.
We performed only basic elicitation on DeepSeek-V3. Given this, it’s difficult to say with confidence whether the results here represent the full extent of the model’s capabilities.
Source-stated
METR says its GPQA results suggest DeepSeek-V3’s performance was not due to memorizing public questions.
This suggests that DeepSeek-V3’s performance on GPQA (and the performance of other frontier models as well) is not the result of memorizing public questions.
Source and contributor context
METR · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
METR
METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: e149b634-edc3-fc72-5cb5-22ecfc764497.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-15T11:08:23.859Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model.
Observation: b75ae1f0a9a0cb5db8ec78cd321e7a4b8fb38a7c4433e596835195785bfd29e9. Source text hash: 5460f4d1a5ce5777d1a348668da5be2f6f0f4a0494825df3c179f9c3ec14fcb1.
Revision and source-check history
Version 1 · 2026-11-15T11:13:26.297Z
METR reports on DeepSeek-V3’s autonomous capabilities
published this evaluation on February 12, 2025. It reports that DeepSeek-V3 performed on par with some earlier in tests of autonomous capabilities, and below others. METR found the model could handle programming and questions but struggled with agency and was sensitive to the scaffolding used to help it act. On RE-Bench, which tests AI R&D tasks, METR found its performance lower than GPT-4o, 3, Claude 3.5 Sonnet, and o1. METR therefore judges DeepSeek-V3’s ability to perform AI R&D autonomously likely insufficient to significantly accelerate algorithmic AI progress. That conclusion is about these evaluations, not a demonstration of real-world performance. METR says it performed only basic elicitation and cannot confidently say whether the results represent the model’s full capabilities. It also reports that the model’s performance on GPQA was not the result of memorizing public questions. METR is not affiliated with DeepSeek and tested a version hosted by a third party.
Original source
· Article
Source details
Original source date: Feb 12, 2025 (2025-02-12T08:00:00.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-15T11:13:26.297Z. First observed: 2026-10-18T03:39:22.959Z. Retrieved: 2026-10-24T01:03:39.430Z.
Limits
Text as retrieved on October 24, 2026; originally published February 12, 2025. says it used basic elicitation and cannot confidently say whether the evaluation represents the full extent of the model’s capabilities. METR conducted its tests on a model hosted by a third party and says it has no affiliation with DeepSeek.
Open question
How much could further elicitation, including different scaffolding or post-training for agency, change DeepSeek-V3’s performance on these tasks?
Source-stated
METR says it found no significant evidence that DeepSeek-V3 has autonomous capabilities beyond those of existing models.
testing it for dual-use autonomous capabilities and failed to find significant evidence for a level of autonomous capabilities beyond those of existing models.
Source-stated
METR reports that DeepSeek-V3’s AI R&D performance was lower than that of the listed models.
DeepSeek-V3’s performance on RE-Bench is lower than that of GPT-4o, Claude 3, Claude 3.5 Sonnet, and o1.
Source-stated
METR says DeepSeek-V3’s autonomous AI R&D abilities are likely insufficient to significantly accelerate algorithmic AI progress.
we think DeepSeek-V3’s abilities to autonomously perform AI R&D are likely insufficient to significantly accelerate the rate of algorithmic AI progress.
Source-stated
METR says its evaluation may not capture the full extent of DeepSeek-V3’s capabilities because it used only basic elicitation.
We performed only basic elicitation on DeepSeek-V3. Given this, it’s difficult to say with confidence whether the results here represent the full extent of the model’s capabilities.
Source-stated
METR says its GPQA results suggest DeepSeek-V3’s performance was not due to memorizing public questions.
This suggests that DeepSeek-V3’s performance on GPQA (and the performance of other frontier models as well) is not the result of memorizing public questions.
Source and contributor context
METR · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
METR
METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: e149b634-edc3-fc72-5cb5-22ecfc764497.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-15T11:08:23.859Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model.
Observation: b75ae1f0a9a0cb5db8ec78cd321e7a4b8fb38a7c4433e596835195785bfd29e9. Source text hash: 5460f4d1a5ce5777d1a348668da5be2f6f0f4a0494825df3c179f9c3ec14fcb1.
uncertain · 2026-11-15T11:08:23.859Z · anthropic/claude-opus-5.5
Publication: e149b634-edc3-fc72-5cb5-22ecfc764497
attribution — pass: This is a single METR evaluation report on DeepSeek-V3. The reading attributes the findings, conclusions and caveats to METR throughout, and correctly notes METR's lack of affiliation with DeepSeek and its use of a third-party host.
METR has no affiliation with DeepSeek and conducted our tests on a model hosted by a 3rd party.
summary — uncertain: Two central terms go unexplained. 'Elicitation' anchors both the caveat and the question; the source defines it as modifying an agent's scaffolding to give it the best chance of succeeding. 'GPQA' appears with no indication that it is a question benchmark, or why memorization would matter. The reading also drops METR's main framing point: open-weight models now match state-of-the-art models from 6 months earlier, after previously lagging much further. Smallest repair: add one-clause glosses for elicitation and GPQA, and one sentence on METR's open-weight gap observation.
Overall, the performance of open-weight models on our tasks is on par with state-of-the-art models from 6 months ago, whereas they had previously lagged significantly further behind.
date — pass: originalPublished is 2025-02-12T08:00Z, and the reading states February 12, 2025, which matches the UTC day. The evidence kind is publisher-page, so no archive disclosure is needed. The retrieval date is given only as a capture note. There is no hindsight framing, and forecasts are not presented as outcomes.
support — uncertain: The claim that RE-Bench performance was lower than GPT-4o and Claude 3 quotes METR's Results section verbatim. However, METR's own summary says DeepSeek-V3 was on par with Claude 3 Opus and very slightly better than GPT-4o. The source is internally inconsistent, and the reading presents only one side. Repair: note the conflict, or say it was comparable to Claude 3 Opus and well below Claude 3.5 Sonnet and o1. The other claims match their excerpts.
DeepSeek-V3’s performance on AI R&D tasks is on-par with the performance of Claude 3 Opus, very slightly better than GPT-4o, and substantially worse than
media — pass: No media was selected and no verified candidates were supplied. Omitting media is appropriate.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METROne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METROne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METR