{"id":"f2ad4395836ccd61fa0dc41d663a8f484a14e50670fbca325b5999e097b613f8","title":"METR reports preliminary GPT-4o evaluation findings","argument":"METR published this report on August 7, 2024, to illustrate its approach to evaluating dangerous autonomous capabilities. METR says GPT-4o, with simple agent scaffolding, performed similarly to human baseliners given 30 minutes per task, though the comparison had large error bars. Agents often completed tasks at a fraction of the cost of humans, but were less effective at turning a larger budget into better performance.\n\nMETR also examined whether failures might be avoidable. It classified 51% of failed runs as spurious and 45% as real capability limitations. After manually correcting one failure pattern, the agent succeeded in 4 out of 10 previously failed runs. METR presents this as evidence that some failures may be overcome with more elicitation. It cautions that human-level performance on these tasks would not necessarily mean a model was dangerous. How results would change with more representative tasks and stronger human comparisons remains unknown.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 24, 2026; originally published August 7, 2024. METR describes the report as informal and based on in-progress research. METR says the tasks were selected for being easy to work with, and that a larger, more diverse set—including more realistic and threat-model-relevant tasks—would be more informative. METR notes that there were at most two human baselines per task, with large variance in completion times, and that human and agent time limits were not directly comparable. The human contractors were not necessarily experts in the relevant domains. ","speaker":{"name":"METR","roleAtSource":"Publisher"},"source":{"title":"Details about METR’s preliminary evaluation of GPT-4o","publisher":"METR","format":"Article","url":"https://metr.org/evaluations/gpt-4o-report/","publishedAt":"2024-08-07T17:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"METR says GPT-4o with simple agent scaffolding performed similarly to human baseliners given 30 minutes per task, while noting large error bars.","excerpt":"The performance was similar to our human baseliners given 30 minutes per task, but there are large error bars on this number.","url":"https://metr.org/evaluations/gpt-4o-report/","relationship":"Source-stated"},{"text":"METR reports that agents often completed tasks at a fraction of the cost of humans, but were less effective at using a larger budget to improve performance.","excerpt":"The agents are often able to complete tasks at a fraction of the cost of humans, but are less effective at turning a larger budget into better performance.","url":"https://metr.org/evaluations/gpt-4o-report/","relationship":"Source-stated"},{"text":"METR classified 51% of 152 failed runs as spurious and 45% as real capability limitations.","excerpt":"68 (45%) of those failures were classified as resulting from a “real” capability limitation, 78 (51%) as spurious failures that seem easily fixed with task-agnostic elicitation effort","url":"https://metr.org/evaluations/gpt-4o-report/","relationship":"Source-stated"},{"text":"After METR manually corrected a particular failure pattern, the agent succeeded in 4 of 10 previously failed runs.","excerpt":"After patching, the agent succeeded in 4 out of 10 previously failed runs.","url":"https://metr.org/evaluations/gpt-4o-report/","relationship":"Source-stated"},{"text":"METR says success on its task set would not necessarily mean a model was dangerous.","excerpt":"achieving human performance on this set of tasks wouldn’t necessarily mean a model was dangerous","url":"https://metr.org/evaluations/gpt-4o-report/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/evaluations/gpt-4o-report/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How would GPT-4o’s results change on a larger, more diverse set of realistic, threat-model-relevant tasks with stronger human comparisons?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"METR reports preliminary GPT-4o evaluation findings","summary":"METR published this report on August 7, 2024, to illustrate its approach to evaluating dangerous autonomous capabilities. METR says GPT-4o, with simple agent scaffolding, performed similarly to human baseliners given 30 minutes per task, though the comparison had large error bars. Agents often completed tasks at a fraction of the cost of humans, but were less effective at turning a larger budget into better performance.\n\nMETR also examined whether failures might be avoidable. It classified 51% of failed runs as spurious and 45% as real capability limitations. After manually correcting one failure pattern, the agent succeeded in 4 out of 10 previously failed runs. METR presents this as evidence that some failures may be overcome with more elicitation. It cautions that human-level performance on these tasks would not necessarily mean a model was dangerous. How results would change with more representative tasks and stronger human comparisons remains unknown.","question":"How would GPT-4o’s results change on a larger, more diverse set of realistic, threat-model-relevant tasks with stronger human comparisons?","claims":[{"text":"METR says GPT-4o with simple agent scaffolding performed similarly to human baseliners given 30 minutes per task, while noting large error bars.","excerpt":"The performance was similar to our human baseliners given 30 minutes per task, but there are large error bars on this number.","support":"source-stated"},{"text":"METR reports that agents often completed tasks at a fraction of the cost of humans, but were less effective at using a larger budget to improve performance.","excerpt":"The agents are often able to complete tasks at a fraction of the cost of humans, but are less effective at turning a larger budget into better performance.","support":"source-stated"},{"text":"METR classified 51% of 152 failed runs as spurious and 45% as real capability limitations.","excerpt":"68 (45%) of those failures were classified as resulting from a “real” capability limitation, 78 (51%) as spurious failures that seem easily fixed with task-agnostic elicitation effort","support":"source-stated"},{"text":"After METR manually corrected a particular failure pattern, the agent succeeded in 4 of 10 previously failed runs.","excerpt":"After patching, the agent succeeded in 4 out of 10 previously failed runs.","support":"source-stated"},{"text":"METR says success on its task set would not necessarily mean a model was dangerous.","excerpt":"achieving human performance on this set of tasks wouldn’t necessarily mean a model was dangerous","support":"source-stated"}],"limitations":["Text as retrieved on October 24, 2026; originally published August 7, 2024.","METR describes the report as informal and based on in-progress research.","METR says the tasks were selected for being easy to work with, and that a larger, more diverse set—including more realistic and threat-model-relevant tasks—would be more informative.","METR notes that there were at most two human baselines per task, with large variance in completion times, and that human and agent time limits were not directly comparable. The human contractors were not necessarily experts in the relevant domains. "],"mediaId":null,"correctionReason":null,"id":"f2ad4395836ccd61fa0dc41d663a8f484a14e50670fbca325b5999e097b613f8","publicationId":"063af6b9-0e12-aff9-dd03-36e7e0250cdc","version":1,"sourceUrl":"https://metr.org/evaluations/gpt-4o-report/","sourceTitle":"Details about METR’s preliminary evaluation of GPT-4o","publisher":"METR","publishedAt":"2024-08-07T17:00:00.000Z","originalPublished":{"value":"2024-08-07T17:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T03:47:22.015Z","createdAt":"2026-11-17T14:09:04.982Z","historical":true,"entityIds":["metr"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/evaluations/gpt-4o-report/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"847fefa075f87db227b14de53bc42519c0b7abcf23791b1b159de0360160e25b","rubricVersion":"unrecorded","provider":"vercel","model":"typesafe-ai/jev","reasonCodes":["bridge_admitted_by_model"],"confidence":0.75,"decidedAt":"2026-10-24T00:43:40.560Z"},"provenance":{"textHash":"8e742922bf84aadac258d588d46e524be6d14652439261ebfaff73143a33d052","observationId":"291c3eb101ba4fbba7c907b97e6a9f5ccb964a7eeb5c718600eef5967151a063","retrievedAt":"2026-10-24T00:46:04.494Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"86db40c8-1032-098d-8e8f-0ae047af9d5d"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"f5f407fb-6cbc-7a81-a7eb-26d93d629ff3"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-17T14:04:46.905Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/metr-reports-preliminary-gpt-4o-evaluation-findings-f2ad4395","formats":{"html":"/readings/metr-reports-preliminary-gpt-4o-evaluation-findings-f2ad4395","markdown":"/readings/metr-reports-preliminary-gpt-4o-evaluation-findings-f2ad4395.md","json":"/readings/metr-reports-preliminary-gpt-4o-evaluation-findings-f2ad4395.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-agent-harness","name":"Agent harness","explanation":"An agent harness is the system that lets an AI model act as an agent: it processes inputs, orchestrates tool calls and returns results. An evaluation harness runs evals end to end, including grading outputs.","referenceUrl":"https://www.anthropic.com/research/demystifying-evals-for-ai-agents"}],"entities":[{"id":"metr","name":"METR","kind":"organization","context":"METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.","referenceUrls":["https://metr.org/"],"path":"/entities/metr","json":"/entities/metr.json","markdown":"/entities/metr.md"},{"id":"time","name":"TIME","kind":"organization","context":"TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.","referenceUrls":["https://time.com/"],"path":"/entities/time","json":"/entities/time.json","markdown":"/entities/time.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"f2ad4395836ccd61fa0dc41d663a8f484a14e50670fbca325b5999e097b613f8","revision":"063af6b9-0e12-aff9-dd03-36e7e0250cdc","revisionKind":"publication-id","title":"METR reports preliminary GPT-4o evaluation findings","url":"https://escapemeridian.com/readings/metr-reports-preliminary-gpt-4o-evaluation-findings-f2ad4395"},"source":{"type":"source","id":"https://metr.org/evaluations/gpt-4o-report/","revision":"8e742922bf84aadac258d588d46e524be6d14652439261ebfaff73143a33d052","revisionKind":"sha256-source-text","title":"Details about METR’s preliminary evaluation of GPT-4o","url":"https://metr.org/evaluations/gpt-4o-report/"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"f2ad4395836ccd61fa0dc41d663a8f484a14e50670fbca325b5999e097b613f8","title":"METR reports preliminary GPT-4o evaluation findings","argument":"METR published this report on August 7, 2024, to illustrate its approach to evaluating dangerous autonomous capabilities. METR says GPT-4o, with simple agent scaffolding, performed similarly to human baseliners given 30 minutes per task, though the comparison had large error bars. Agents often completed tasks at a fraction of the cost of humans, but were less effective at turning a larger budget into better performance.\n\nMETR also examined whether failures might be avoidable. It classified 51% of failed runs as spurious and 45% as real capability limitations. After manually correcting one failure pattern, the agent succeeded in 4 out of 10 previously failed runs. METR presents this as evidence that some failures may be overcome with more elicitation. It cautions that human-level performance on these tasks would not necessarily mean a model was dangerous. How results would change with more representative tasks and stronger human comparisons remains unknown.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 24, 2026; originally published August 7, 2024. METR describes the report as informal and based on in-progress research. METR says the tasks were selected for being easy to work with, and that a larger, more diverse set—including more realistic and threat-model-relevant tasks—would be more informative. METR notes that there were at most two human baselines per task, with large variance in completion times, and that human and agent time limits were not directly comparable. The human contractors were not necessarily experts in the relevant domains. ","speaker":{"name":"METR","roleAtSource":"Publisher"},"source":{"title":"Details about METR’s preliminary evaluation of GPT-4o","publisher":"METR","format":"Article","url":"https://metr.org/evaluations/gpt-4o-report/","publishedAt":"2024-08-07T17:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"METR says GPT-4o with simple agent scaffolding performed similarly to human baseliners given 30 minutes per task, while noting large error bars.","excerpt":"The performance was similar to our human baseliners given 30 minutes per task, but there are large error bars on this number.","url":"https://metr.org/evaluations/gpt-4o-report/","relationship":"Source-stated"},{"text":"METR reports that agents often completed tasks at a fraction of the cost of humans, but were less effective at using a larger budget to improve performance.","excerpt":"The agents are often able to complete tasks at a fraction of the cost of humans, but are less effective at turning a larger budget into better performance.","url":"https://metr.org/evaluations/gpt-4o-report/","relationship":"Source-stated"},{"text":"METR classified 51% of 152 failed runs as spurious and 45% as real capability limitations.","excerpt":"68 (45%) of those failures were classified as resulting from a “real” capability limitation, 78 (51%) as spurious failures that seem easily fixed with task-agnostic elicitation effort","url":"https://metr.org/evaluations/gpt-4o-report/","relationship":"Source-stated"},{"text":"After METR manually corrected a particular failure pattern, the agent succeeded in 4 of 10 previously failed runs.","excerpt":"After patching, the agent succeeded in 4 out of 10 previously failed runs.","url":"https://metr.org/evaluations/gpt-4o-report/","relationship":"Source-stated"},{"text":"METR says success on its task set would not necessarily mean a model was dangerous.","excerpt":"achieving human performance on this set of tasks wouldn’t necessarily mean a model was dangerous","url":"https://metr.org/evaluations/gpt-4o-report/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/evaluations/gpt-4o-report/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How would GPT-4o’s results change on a larger, more diverse set of realistic, threat-model-relevant tasks with stronger human comparisons?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"METR reports preliminary GPT-4o evaluation findings","summary":"METR published this report on August 7, 2024, to illustrate its approach to evaluating dangerous autonomous capabilities. METR says GPT-4o, with simple agent scaffolding, performed similarly to human baseliners given 30 minutes per task, though the comparison had large error bars. Agents often completed tasks at a fraction of the cost of humans, but were less effective at turning a larger budget into better performance.\n\nMETR also examined whether failures might be avoidable. It classified 51% of failed runs as spurious and 45% as real capability limitations. After manually correcting one failure pattern, the agent succeeded in 4 out of 10 previously failed runs. METR presents this as evidence that some failures may be overcome with more elicitation. It cautions that human-level performance on these tasks would not necessarily mean a model was dangerous. How results would change with more representative tasks and stronger human comparisons remains unknown.","question":"How would GPT-4o’s results change on a larger, more diverse set of realistic, threat-model-relevant tasks with stronger human comparisons?","claims":[{"text":"METR says GPT-4o with simple agent scaffolding performed similarly to human baseliners given 30 minutes per task, while noting large error bars.","excerpt":"The performance was similar to our human baseliners given 30 minutes per task, but there are large error bars on this number.","support":"source-stated"},{"text":"METR reports that agents often completed tasks at a fraction of the cost of humans, but were less effective at using a larger budget to improve performance.","excerpt":"The agents are often able to complete tasks at a fraction of the cost of humans, but are less effective at turning a larger budget into better performance.","support":"source-stated"},{"text":"METR classified 51% of 152 failed runs as spurious and 45% as real capability limitations.","excerpt":"68 (45%) of those failures were classified as resulting from a “real” capability limitation, 78 (51%) as spurious failures that seem easily fixed with task-agnostic elicitation effort","support":"source-stated"},{"text":"After METR manually corrected a particular failure pattern, the agent succeeded in 4 of 10 previously failed runs.","excerpt":"After patching, the agent succeeded in 4 out of 10 previously failed runs.","support":"source-stated"},{"text":"METR says success on its task set would not necessarily mean a model was dangerous.","excerpt":"achieving human performance on this set of tasks wouldn’t necessarily mean a model was dangerous","support":"source-stated"}],"limitations":["Text as retrieved on October 24, 2026; originally published August 7, 2024.","METR describes the report as informal and based on in-progress research.","METR says the tasks were selected for being easy to work with, and that a larger, more diverse set—including more realistic and threat-model-relevant tasks—would be more informative.","METR notes that there were at most two human baselines per task, with large variance in completion times, and that human and agent time limits were not directly comparable. The human contractors were not necessarily experts in the relevant domains. "],"mediaId":null,"correctionReason":null,"id":"f2ad4395836ccd61fa0dc41d663a8f484a14e50670fbca325b5999e097b613f8","publicationId":"063af6b9-0e12-aff9-dd03-36e7e0250cdc","version":1,"sourceUrl":"https://metr.org/evaluations/gpt-4o-report/","sourceTitle":"Details about METR’s preliminary evaluation of GPT-4o","publisher":"METR","publishedAt":"2024-08-07T17:00:00.000Z","originalPublished":{"value":"2024-08-07T17:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T03:47:22.015Z","createdAt":"2026-11-17T14:09:04.982Z","historical":true,"entityIds":["metr"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/evaluations/gpt-4o-report/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"847fefa075f87db227b14de53bc42519c0b7abcf23791b1b159de0360160e25b","rubricVersion":"unrecorded","provider":"vercel","model":"typesafe-ai/jev","reasonCodes":["bridge_admitted_by_model"],"confidence":0.75,"decidedAt":"2026-10-24T00:43:40.560Z"},"provenance":{"textHash":"8e742922bf84aadac258d588d46e524be6d14652439261ebfaff73143a33d052","observationId":"291c3eb101ba4fbba7c907b97e6a9f5ccb964a7eeb5c718600eef5967151a063","retrievedAt":"2026-10-24T00:46:04.494Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"86db40c8-1032-098d-8e8f-0ae047af9d5d"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"f5f407fb-6cbc-7a81-a7eb-26d93d629ff3"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-17T14:04:46.905Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"063af6b9-0e12-aff9-dd03-36e7e0250cdc","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading consistently attributes findings to METR, the publisher of this single report page about its preliminary GPT-4o evaluation. Claims are framed as what METR says or reports, not as established fact.","excerpt":"This page provides additional details about METR’s preliminary evaluation of GPT-4o"},{"dimension":"summary","status":"uncertain","explanation":"The summary is faithful and leads with the actor and findings. Two central terms are unexplained, though: 'spurious' failures, which the source defines as mistakes that seem easy to fix in task-agnostic ways, and 'elicitation', the iterative modification of an agent to improve its performance. The second paragraph's meaning depends on both, and only agent scaffolding appears in the definitions. Smallest repair: add a plain-language gloss for each term, or term-library entries for spurious failure and elicitation.","excerpt":"We refer to the process of iteratively modifying an agent in an attempt to improve its performance as elicitation."},{"dimension":"date","status":"pass","explanation":"The displayed date of August 7, 2024 matches the UTC calendar day of originalPublished. The reading uses nothing after that date and frames nothing with hindsight. The page is a publisher page, not an archive copy, so no archive disclosure is required. The embedded 7/17/2024 timestamp predates publication and is not relied on.","excerpt":null},{"dimension":"support","status":"pass","explanation":"Each claim matches the source: the 30-minute human comparison with large error bars, the cost result, the 51%/45% split of 152 classified failures, the 4/10 patching result, and the caveat that human-level performance would not necessarily mean danger. The limitations and the question follow METR's own stated limitations. No unsupported benefits or costs are implied.","excerpt":"A larger and more diverse set of tasks would be more informative."},{"dimension":"media","status":"pass","explanation":"No media was selected, and no verified media candidates were supplied, so no media mismatch exists.","excerpt":null}],"createdAt":"2026-11-17T14:04:46.905Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}