· Aug 7, 2024
METR reports preliminary GPT-4o evaluation findings
Markdown · JSON · Original source
published this report on August 7, 2024, to illustrate its approach to evaluating dangerous autonomous capabilities. METR says GPT-4o, with simple , performed similarly to human baseliners given 30 minutes per task, though the comparison had large error bars. Agents often completed tasks at a fraction of the cost of humans, but were less effective at turning a larger budget into better performance. METR also examined whether failures might be avoidable. It classified 51% of failed runs as spurious and 45% as real capability limitations. After manually correcting one failure pattern, the agent succeeded in 4 out of 10 previously failed runs. METR presents this as evidence that some failures may be overcome with more elicitation. It cautions that human-level performance on these tasks would not necessarily mean a model was dangerous. How results would change with more representative tasks and stronger human comparisons remains unknown.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
· Article
Source details
Original source date: Aug 7, 2024 (2024-08-07T17:00:00.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-17T14:09:04.982Z. First observed: 2026-10-18T03:47:22.015Z. Retrieved: 2026-10-24T00:46:04.494Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
Text as retrieved on October 24, 2026; originally published August 7, 2024. describes the report as informal and based on in-progress research. METR says the tasks were selected for being easy to work with, and that a larger, more diverse set—including more realistic and threat-model-relevant tasks—would be more informative. METR notes that there were at most two human baselines per task, with large variance in completion times, and that human and agent limits were not directly comparable. The human contractors were not necessarily experts in the relevant domains.
Open question
How would GPT-4o’s results change on a larger, more diverse set of realistic, threat-model-relevant tasks with stronger human comparisons?
Source-stated
METR says GPT-4o with simple agent scaffolding performed similarly to human baseliners given 30 minutes per task, while noting large error bars.
The performance was similar to our human baseliners given 30 minutes per task, but there are large error bars on this number.
Source-stated
METR reports that agents often completed tasks at a fraction of the cost of humans, but were less effective at using a larger budget to improve performance.
The agents are often able to complete tasks at a fraction of the cost of humans, but are less effective at turning a larger budget into better performance.
Source-stated
METR classified 51% of 152 failed runs as spurious and 45% as real capability limitations.
68 (45%) of those failures were classified as resulting from a “real” capability limitation, 78 (51%) as spurious failures that seem easily fixed with task-agnostic elicitation effort
Source-stated
After METR manually corrected a particular failure pattern, the agent succeeded in 4 of 10 previously failed runs.
After patching, the agent succeeded in 4 out of 10 previously failed runs.
Source-stated
METR says success on its task set would not necessarily mean a model was dangerous.
achieving human performance on this set of tasks wouldn’t necessarily mean a model was dangerous
Source and contributor context
METR · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Agent harness
An agent harness is the system that lets an AI model act as an agent: it processes inputs, orchestrates tool calls and returns results. An evaluation harness runs evals end to end, including grading outputs.
METR
METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 063af6b9-0e12-aff9-dd03-36e7e0250cdc.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-17T14:04:46.905Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model.
Observation: 291c3eb101ba4fbba7c907b97e6a9f5ccb964a7eeb5c718600eef5967151a063. Source text hash: 8e742922bf84aadac258d588d46e524be6d14652439261ebfaff73143a33d052.
Revision and source-check history
Version 1 · 2026-11-17T14:09:04.982Z
METR reports preliminary GPT-4o evaluation findings
published this report on August 7, 2024, to illustrate its approach to evaluating dangerous autonomous capabilities. METR says GPT-4o, with simple , performed similarly to human baseliners given 30 minutes per task, though the comparison had large error bars. Agents often completed tasks at a fraction of the cost of humans, but were less effective at turning a larger budget into better performance. METR also examined whether failures might be avoidable. It classified 51% of failed runs as spurious and 45% as real capability limitations. After manually correcting one failure pattern, the agent succeeded in 4 out of 10 previously failed runs. METR presents this as evidence that some failures may be overcome with more elicitation. It cautions that human-level performance on these tasks would not necessarily mean a model was dangerous. How results would change with more representative tasks and stronger human comparisons remains unknown.
Original source
· Article
Source details
Original source date: Aug 7, 2024 (2024-08-07T17:00:00.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-17T14:09:04.982Z. First observed: 2026-10-18T03:47:22.015Z. Retrieved: 2026-10-24T00:46:04.494Z.
Limits
Text as retrieved on October 24, 2026; originally published August 7, 2024. describes the report as informal and based on in-progress research. METR says the tasks were selected for being easy to work with, and that a larger, more diverse set—including more realistic and threat-model-relevant tasks—would be more informative. METR notes that there were at most two human baselines per task, with large variance in completion times, and that human and agent limits were not directly comparable. The human contractors were not necessarily experts in the relevant domains.
Open question
How would GPT-4o’s results change on a larger, more diverse set of realistic, threat-model-relevant tasks with stronger human comparisons?
Source-stated
METR says GPT-4o with simple agent scaffolding performed similarly to human baseliners given 30 minutes per task, while noting large error bars.
The performance was similar to our human baseliners given 30 minutes per task, but there are large error bars on this number.
Source-stated
METR reports that agents often completed tasks at a fraction of the cost of humans, but were less effective at using a larger budget to improve performance.
The agents are often able to complete tasks at a fraction of the cost of humans, but are less effective at turning a larger budget into better performance.
Source-stated
METR classified 51% of 152 failed runs as spurious and 45% as real capability limitations.
68 (45%) of those failures were classified as resulting from a “real” capability limitation, 78 (51%) as spurious failures that seem easily fixed with task-agnostic elicitation effort
Source-stated
After METR manually corrected a particular failure pattern, the agent succeeded in 4 of 10 previously failed runs.
After patching, the agent succeeded in 4 out of 10 previously failed runs.
Source-stated
METR says success on its task set would not necessarily mean a model was dangerous.
achieving human performance on this set of tasks wouldn’t necessarily mean a model was dangerous
Source and contributor context
METR · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Agent harness
An agent harness is the system that lets an AI model act as an agent: it processes inputs, orchestrates tool calls and returns results. An evaluation harness runs evals end to end, including grading outputs.
METR
METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 063af6b9-0e12-aff9-dd03-36e7e0250cdc.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-17T14:04:46.905Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model.
Observation: 291c3eb101ba4fbba7c907b97e6a9f5ccb964a7eeb5c718600eef5967151a063. Source text hash: 8e742922bf84aadac258d588d46e524be6d14652439261ebfaff73143a33d052.
uncertain · 2026-11-17T14:04:46.905Z · anthropic/claude-opus-5.5
Publication: 063af6b9-0e12-aff9-dd03-36e7e0250cdc
attribution — pass: The reading consistently attributes findings to METR, the publisher of this single report page about its preliminary GPT-4o evaluation. Claims are framed as what METR says or reports, not as established fact.
This page provides additional details about METR’s preliminary evaluation of GPT-4o
summary — uncertain: The summary is faithful and leads with the actor and findings. Two central terms are unexplained, though: 'spurious' failures, which the source defines as mistakes that seem easy to fix in task-agnostic ways, and 'elicitation', the iterative modification of an agent to improve its performance. The second paragraph's meaning depends on both, and only agent scaffolding appears in the definitions. Smallest repair: add a plain-language gloss for each term, or term-library entries for spurious failure and elicitation.
We refer to the process of iteratively modifying an agent in an attempt to improve its performance as elicitation.
date — pass: The displayed date of August 7, 2024 matches the UTC calendar day of originalPublished. The reading uses nothing after that date and frames nothing with hindsight. The page is a publisher page, not an archive copy, so no archive disclosure is required. The embedded 7/17/2024 timestamp predates publication and is not relied on.
support — pass: Each claim matches the source: the 30-minute human comparison with large error bars, the cost result, the 51%/45% split of 152 classified failures, the 4/10 patching result, and the caveat that human-level performance would not necessarily mean danger. The limitations and the question follow METR's own stated limitations. No unsupported benefits or costs are implied.
A larger and more diverse set of tasks would be more informative.
media — pass: No media was selected, and no verified media candidates were supplied, so no media mismatch exists.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRAn agent harness is the system that lets an AI model act as an agent: it processes inputs, orchestrates tool calls and returns results. An evaluation harness runs evals end to end, including grading outputs.
Definition referenceMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRAn agent harness is the system that lets an AI model act as an agent: it processes inputs, orchestrates tool calls and returns results. An evaluation harness runs evals end to end, including grading outputs.
Definition referenceMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIME