METR reports DeepSeek-R1’s results on autonomous capabilities

METR reports DeepSeek-R1’s results on autonomous capabilities

METR published this evaluation on March 5, 2025. It reports that DeepSeek-R1 performed slightly better than o1-preview on its general autonomy tasks, but that it found no significant evidence of autonomous capabilities beyond existing models. On RE-Bench, DeepSeek-R1 performed lower than GPT-4o, Claude 3, Claude 3.5 Sonnet and o1. METR concludes that the model’s abilities to autonomously perform AI R&D are likely insufficient to significantly accelerate the rate of algorithmic AI progress. It also describes recurring problems with instruction following and recovering from mistakes. These results concern METR’s task suites: the report says its tasks may be too artificial to capture real-world demands, and that moderate elicitation makes it difficult to know whether the results represent the model’s full capabilities. METR says further elicitation might improve performance, but how much it would change the evaluation results remains unknown.

Publication: 5402502d-df3f-cb59-8c9a-1791490ce534

attribution — pass: The reading consistently attributes the findings, conclusions and caveats to METR, the publisher of this single evaluation report. It keeps METR's assessments as METR's statements rather than established facts.

We performed two evaluations of DeepSeek-R1 testing for dual-use autonomous capabilities

summary — uncertain: Central terms are left unexplained, and no definitions are supplied. 'Elicitation' anchors both the closing caveat and the question, yet a general reader is not told it means adjusting the scaffolding around a model to give it its best chance on the tasks. 'RE-Bench', 'AI R&D', 'autonomous capabilities' and 'o1-preview' are also unexplained. Smallest repair: add short plain-language glosses or term-library tooltips. The reading also omits METR's headline significance: open-weight models are on par with state-of-the-art models from 7 months earlier but 5-28x cheaper. Restoring that clause would preserve why the report matters.

We call the process of modifying the scaffolding around an agent to give it the best shot at succeeding at our task suite “elicitation”

date — pass: The reading states March 5, 2025, which is the UTC calendar day of evidence.originalPublished, at day precision. Nothing later is presented as known, and there is no hindsight framing. METR's judgements about future elicitation are kept as METR's statements. The page is a publisher page, so no archive disclosure is needed.

support — pass: Each claim matches the source. This includes: no significant evidence of capabilities beyond existing models; lower RE-Bench performance than GPT-4o, Claude 3, Claude 3.5 Sonnet and o1; AI R&D abilities likely insufficient to accelerate progress; repeated failure modes in instruction following and recovery from mistakes; and the artificiality and elicitation caveats. The source's earlier line calls R1 comparable to GPT-4o on RE-Bench, while its results section says lower. That inconsistency is the source's, and the reading quotes the results section accurately. The question follows from METR's stated uncertainty.

In our experience eliciting DeepSeek-R1 and inspecting its outputs, there are a few common failure modes we noticed coming up repeatedly

media — pass: No media is selected, and no verified media candidates were supplied. Omitting media is appropriate.