{"id":"2040a9becbc0291768af785ba162ee97d9336b2038118269de831efabfc4c35f","title":"METR reports a benchmark of AI agents on ML research engineering tasks","argument":"METR published RE-Bench on November 22, 2024, comparing people and AI agents on machine learning research engineering tasks. The benchmark gives participants a computer, a scoring function and a fixed time budget to work on tasks such as improving code runtime or designing model architectures.\n\nMETR reports that agents using Claude 3.5 Sonnet and o1-preview scored higher on average than human experts with short time budgets, while human scores improved faster as budgets grew. At a 32-hour budget, the average human score was almost twice that of the best AI agent. METR says the agents cost several times less per hour and suggests better scaffolding and orchestration could improve performance for a given cost.\n\nMETR also reports that agents often struggled to respond to new information or build on earlier progress. The authors stress that RE-Bench has few environments, clear objectives and faster feedback than much real-world ML research. They say it remains unclear how well research problems can be packaged into small tasks for models to iterate on. METR hopes RE-Bench will help evaluate dangerous levels of autonomous AI R&D capabilities, which it identifies as a capability of concern in AI safety policies.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 23, 2026; originally published November 22, 2024. METR notes that the benchmark has a small number of environments and that its clear goals and faster feedback differ from real-world ML research. METR says better scaffolding and orchestration may improve agent performance; the cited results do not establish how much improvement would follow. The results concern benchmark tasks and do not establish agents’ performance across real-world ML R&D.","speaker":{"name":"METR","roleAtSource":"Publisher"},"source":{"title":"Evaluating frontier AI R&D capabilities of language model agents against human experts","publisher":"METR","format":"Article","url":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","publishedAt":"2024-11-22T08:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"METR released RE-Bench to compare human experts and frontier model agents on machine learning research engineering tasks.","excerpt":"We’re releasing RE-Bench, a new benchmark for measuring the performance of humans and frontier model agents on ML research engineering tasks.","url":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","relationship":"Source-stated"},{"text":"METR reports that agents scored higher on average than human experts with short time budgets, while human scores improved faster at longer budgets.","excerpt":"When both have short total time budgets (e.g. 2 hours), AI agents that use frontier models Claude-3.5-Sonnet (new) and o1-preview score higher than our human experts on average. However, as we consider longer time budgets, human scores improve at a faster rate than AI agents’.","url":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","relationship":"Source-stated"},{"text":"At a 32-hour time budget, the average human score was almost twice that of the best AI agent.","excerpt":"At a 32-hour time budget, the average human score is almost twice that of the best AI agent.","url":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","relationship":"Source-stated"},{"text":"METR reports that the agents cost less per hour than human experts and suggests that better scaffolding and orchestration could improve agent performance for a given cost.","excerpt":"On the other hand, the hourly cost of running these AI agents is several times cheaper than humans. This suggests that better scaffolding and orchestration could increase model performance substantially for a given cost.","url":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","relationship":"Source-stated"},{"text":"METR says agents often struggle to respond to new information or build on progress over time.","excerpt":"The median AI agent attempt (when we don’t do “best-of-k”) makes very little progress in most environments, and we often observe the agents failing to react appropriately to novel information or struggling to build on their progress over time.","url":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","relationship":"Source-stated"},{"text":"METR says real-world ML research often has slower feedback and less clearly defined goals, leaving uncertainty about whether research can be packaged into small tasks for models to iterate on.","excerpt":"However, real-world ML R&D work often has much slower feedback loops and less clearly-defined goals than in our environments, and it’s unclear to what extent research problems can be packaged into small tasks that models can iterate on effectively.","url":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","relationship":"Source-stated"},{"text":"METR presents RE-Bench as a possible tool for evaluations of dangerous levels of autonomous AI R&D capabilities.","excerpt":"We hope RE-Bench and the methodology we’ve developed will be useful in creating evaluations for dangerous levels of autonomous AI R&D capabilities.","url":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well do RE-Bench results predict agents’ performance on real-world ML research with less clearly defined goals and slower feedback?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"METR reports a benchmark of AI agents on ML research engineering tasks","summary":"METR published RE-Bench on November 22, 2024, comparing people and AI agents on machine learning research engineering tasks. The benchmark gives participants a computer, a scoring function and a fixed time budget to work on tasks such as improving code runtime or designing model architectures.\n\nMETR reports that agents using Claude 3.5 Sonnet and o1-preview scored higher on average than human experts with short time budgets, while human scores improved faster as budgets grew. At a 32-hour budget, the average human score was almost twice that of the best AI agent. METR says the agents cost several times less per hour and suggests better scaffolding and orchestration could improve performance for a given cost.\n\nMETR also reports that agents often struggled to respond to new information or build on earlier progress. The authors stress that RE-Bench has few environments, clear objectives and faster feedback than much real-world ML research. They say it remains unclear how well research problems can be packaged into small tasks for models to iterate on. METR hopes RE-Bench will help evaluate dangerous levels of autonomous AI R&D capabilities, which it identifies as a capability of concern in AI safety policies.","question":"How well do RE-Bench results predict agents’ performance on real-world ML research with less clearly defined goals and slower feedback?","claims":[{"text":"METR released RE-Bench to compare human experts and frontier model agents on machine learning research engineering tasks.","excerpt":"We’re releasing RE-Bench, a new benchmark for measuring the performance of humans and frontier model agents on ML research engineering tasks.","support":"source-stated"},{"text":"METR reports that agents scored higher on average than human experts with short time budgets, while human scores improved faster at longer budgets.","excerpt":"When both have short total time budgets (e.g. 2 hours), AI agents that use frontier models Claude-3.5-Sonnet (new) and o1-preview score higher than our human experts on average. However, as we consider longer time budgets, human scores improve at a faster rate than AI agents’.","support":"source-stated"},{"text":"At a 32-hour time budget, the average human score was almost twice that of the best AI agent.","excerpt":"At a 32-hour time budget, the average human score is almost twice that of the best AI agent.","support":"source-stated"},{"text":"METR reports that the agents cost less per hour than human experts and suggests that better scaffolding and orchestration could improve agent performance for a given cost.","excerpt":"On the other hand, the hourly cost of running these AI agents is several times cheaper than humans. This suggests that better scaffolding and orchestration could increase model performance substantially for a given cost.","support":"source-stated"},{"text":"METR says agents often struggle to respond to new information or build on progress over time.","excerpt":"The median AI agent attempt (when we don’t do “best-of-k”) makes very little progress in most environments, and we often observe the agents failing to react appropriately to novel information or struggling to build on their progress over time.","support":"source-stated"},{"text":"METR says real-world ML research often has slower feedback and less clearly defined goals, leaving uncertainty about whether research can be packaged into small tasks for models to iterate on.","excerpt":"However, real-world ML R&D work often has much slower feedback loops and less clearly-defined goals than in our environments, and it’s unclear to what extent research problems can be packaged into small tasks that models can iterate on effectively.","support":"source-stated"},{"text":"METR presents RE-Bench as a possible tool for evaluations of dangerous levels of autonomous AI R&D capabilities.","excerpt":"We hope RE-Bench and the methodology we’ve developed will be useful in creating evaluations for dangerous levels of autonomous AI R&D capabilities.","support":"source-stated"}],"limitations":["Text as retrieved on October 23, 2026; originally published November 22, 2024.","METR notes that the benchmark has a small number of environments and that its clear goals and faster feedback differ from real-world ML research.","METR says better scaffolding and orchestration may improve agent performance; the cited results do not establish how much improvement would follow.","The results concern benchmark tasks and do not establish agents’ performance across real-world ML R&D."],"mediaId":null,"correctionReason":null,"id":"2040a9becbc0291768af785ba162ee97d9336b2038118269de831efabfc4c35f","publicationId":"156ced09-5bc4-3d9d-3e04-00f1d44c1906","version":1,"sourceUrl":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","sourceTitle":"Evaluating frontier AI R&D capabilities of language model agents against human experts","publisher":"METR","publishedAt":"2024-11-22T08:00:00.000Z","originalPublished":{"value":"2024-11-22T08:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T03:43:19.850Z","createdAt":"2026-11-16T10:58:22.973Z","historical":true,"entityIds":["metr"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"049b017a51baa9c662a6db45bb211493e111b53cd1cfa87b9d75db664a85b03e","rubricVersion":"unrecorded","provider":"vercel","model":"typesafe-ai/jev","reasonCodes":["bridge_admitted_by_model"],"confidence":0.6,"decidedAt":"2026-10-23T23:57:42.096Z"},"provenance":{"textHash":"e4c59bfca9d4ec2749a352d2bb84f65412b0c263b44a2997d62b59ce1d4aa742","observationId":"81b12e818d59f4eeea6ab08acd215d5a54c85d83be5bfd727a232bc3b026b71a","retrievedAt":"2026-10-23T23:59:37.953Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"8a8d6cd5-80bb-1421-321d-486a9f03dcb0"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"dc459273-22b8-f816-55e1-4c4aabe3dbcf"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-16T10:53:43.858Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/metr-reports-a-benchmark-of-ai-agents-on-ml-research-2040a9be","formats":{"html":"/readings/metr-reports-a-benchmark-of-ai-agents-on-ml-research-2040a9be","markdown":"/readings/metr-reports-a-benchmark-of-ai-agents-on-ml-research-2040a9be.md","json":"/readings/metr-reports-a-benchmark-of-ai-agents-on-ml-research-2040a9be.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-ai-agent","name":"AI agent","explanation":"An AI system that chooses and carries out actions toward a goal, often using software tools.","referenceUrl":"https://www.ibm.com/think/topics/ai-agents"},{"id":"term-machine-learning","name":"Machine learning","explanation":"Methods that let computers learn patterns from examples instead of following rules written by hand.","referenceUrl":"https://developers.google.com/machine-learning/glossary#machine-learning"}],"entities":[{"id":"metr","name":"METR","kind":"organization","context":"METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.","referenceUrls":["https://metr.org/"],"path":"/entities/metr","json":"/entities/metr.json","markdown":"/entities/metr.md"},{"id":"time","name":"TIME","kind":"organization","context":"TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.","referenceUrls":["https://time.com/"],"path":"/entities/time","json":"/entities/time.json","markdown":"/entities/time.md"},{"id":"claude","name":"Claude","kind":"model","context":"Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.","referenceUrls":["https://claude.com/product/overview","https://www.anthropic.com/news","https://www.anthropic.com/research/team/economics"],"path":"/entities/claude","json":"/entities/claude.json","markdown":"/entities/claude.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"2040a9becbc0291768af785ba162ee97d9336b2038118269de831efabfc4c35f","revision":"156ced09-5bc4-3d9d-3e04-00f1d44c1906","revisionKind":"publication-id","title":"METR reports a benchmark of AI agents on ML research engineering tasks","url":"https://escapemeridian.com/readings/metr-reports-a-benchmark-of-ai-agents-on-ml-research-2040a9be"},"source":{"type":"source","id":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","revision":"e4c59bfca9d4ec2749a352d2bb84f65412b0c263b44a2997d62b59ce1d4aa742","revisionKind":"sha256-source-text","title":"Evaluating frontier AI R&D capabilities of language model agents against human experts","url":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"2040a9becbc0291768af785ba162ee97d9336b2038118269de831efabfc4c35f","title":"METR reports a benchmark of AI agents on ML research engineering tasks","argument":"METR published RE-Bench on November 22, 2024, comparing people and AI agents on machine learning research engineering tasks. The benchmark gives participants a computer, a scoring function and a fixed time budget to work on tasks such as improving code runtime or designing model architectures.\n\nMETR reports that agents using Claude 3.5 Sonnet and o1-preview scored higher on average than human experts with short time budgets, while human scores improved faster as budgets grew. At a 32-hour budget, the average human score was almost twice that of the best AI agent. METR says the agents cost several times less per hour and suggests better scaffolding and orchestration could improve performance for a given cost.\n\nMETR also reports that agents often struggled to respond to new information or build on earlier progress. The authors stress that RE-Bench has few environments, clear objectives and faster feedback than much real-world ML research. They say it remains unclear how well research problems can be packaged into small tasks for models to iterate on. METR hopes RE-Bench will help evaluate dangerous levels of autonomous AI R&D capabilities, which it identifies as a capability of concern in AI safety policies.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 23, 2026; originally published November 22, 2024. METR notes that the benchmark has a small number of environments and that its clear goals and faster feedback differ from real-world ML research. METR says better scaffolding and orchestration may improve agent performance; the cited results do not establish how much improvement would follow. The results concern benchmark tasks and do not establish agents’ performance across real-world ML R&D.","speaker":{"name":"METR","roleAtSource":"Publisher"},"source":{"title":"Evaluating frontier AI R&D capabilities of language model agents against human experts","publisher":"METR","format":"Article","url":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","publishedAt":"2024-11-22T08:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"METR released RE-Bench to compare human experts and frontier model agents on machine learning research engineering tasks.","excerpt":"We’re releasing RE-Bench, a new benchmark for measuring the performance of humans and frontier model agents on ML research engineering tasks.","url":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","relationship":"Source-stated"},{"text":"METR reports that agents scored higher on average than human experts with short time budgets, while human scores improved faster at longer budgets.","excerpt":"When both have short total time budgets (e.g. 2 hours), AI agents that use frontier models Claude-3.5-Sonnet (new) and o1-preview score higher than our human experts on average. However, as we consider longer time budgets, human scores improve at a faster rate than AI agents’.","url":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","relationship":"Source-stated"},{"text":"At a 32-hour time budget, the average human score was almost twice that of the best AI agent.","excerpt":"At a 32-hour time budget, the average human score is almost twice that of the best AI agent.","url":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","relationship":"Source-stated"},{"text":"METR reports that the agents cost less per hour than human experts and suggests that better scaffolding and orchestration could improve agent performance for a given cost.","excerpt":"On the other hand, the hourly cost of running these AI agents is several times cheaper than humans. This suggests that better scaffolding and orchestration could increase model performance substantially for a given cost.","url":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","relationship":"Source-stated"},{"text":"METR says agents often struggle to respond to new information or build on progress over time.","excerpt":"The median AI agent attempt (when we don’t do “best-of-k”) makes very little progress in most environments, and we often observe the agents failing to react appropriately to novel information or struggling to build on their progress over time.","url":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","relationship":"Source-stated"},{"text":"METR says real-world ML research often has slower feedback and less clearly defined goals, leaving uncertainty about whether research can be packaged into small tasks for models to iterate on.","excerpt":"However, real-world ML R&D work often has much slower feedback loops and less clearly-defined goals than in our environments, and it’s unclear to what extent research problems can be packaged into small tasks that models can iterate on effectively.","url":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","relationship":"Source-stated"},{"text":"METR presents RE-Bench as a possible tool for evaluations of dangerous levels of autonomous AI R&D capabilities.","excerpt":"We hope RE-Bench and the methodology we’ve developed will be useful in creating evaluations for dangerous levels of autonomous AI R&D capabilities.","url":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well do RE-Bench results predict agents’ performance on real-world ML research with less clearly defined goals and slower feedback?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"METR reports a benchmark of AI agents on ML research engineering tasks","summary":"METR published RE-Bench on November 22, 2024, comparing people and AI agents on machine learning research engineering tasks. The benchmark gives participants a computer, a scoring function and a fixed time budget to work on tasks such as improving code runtime or designing model architectures.\n\nMETR reports that agents using Claude 3.5 Sonnet and o1-preview scored higher on average than human experts with short time budgets, while human scores improved faster as budgets grew. At a 32-hour budget, the average human score was almost twice that of the best AI agent. METR says the agents cost several times less per hour and suggests better scaffolding and orchestration could improve performance for a given cost.\n\nMETR also reports that agents often struggled to respond to new information or build on earlier progress. The authors stress that RE-Bench has few environments, clear objectives and faster feedback than much real-world ML research. They say it remains unclear how well research problems can be packaged into small tasks for models to iterate on. METR hopes RE-Bench will help evaluate dangerous levels of autonomous AI R&D capabilities, which it identifies as a capability of concern in AI safety policies.","question":"How well do RE-Bench results predict agents’ performance on real-world ML research with less clearly defined goals and slower feedback?","claims":[{"text":"METR released RE-Bench to compare human experts and frontier model agents on machine learning research engineering tasks.","excerpt":"We’re releasing RE-Bench, a new benchmark for measuring the performance of humans and frontier model agents on ML research engineering tasks.","support":"source-stated"},{"text":"METR reports that agents scored higher on average than human experts with short time budgets, while human scores improved faster at longer budgets.","excerpt":"When both have short total time budgets (e.g. 2 hours), AI agents that use frontier models Claude-3.5-Sonnet (new) and o1-preview score higher than our human experts on average. However, as we consider longer time budgets, human scores improve at a faster rate than AI agents’.","support":"source-stated"},{"text":"At a 32-hour time budget, the average human score was almost twice that of the best AI agent.","excerpt":"At a 32-hour time budget, the average human score is almost twice that of the best AI agent.","support":"source-stated"},{"text":"METR reports that the agents cost less per hour than human experts and suggests that better scaffolding and orchestration could improve agent performance for a given cost.","excerpt":"On the other hand, the hourly cost of running these AI agents is several times cheaper than humans. This suggests that better scaffolding and orchestration could increase model performance substantially for a given cost.","support":"source-stated"},{"text":"METR says agents often struggle to respond to new information or build on progress over time.","excerpt":"The median AI agent attempt (when we don’t do “best-of-k”) makes very little progress in most environments, and we often observe the agents failing to react appropriately to novel information or struggling to build on their progress over time.","support":"source-stated"},{"text":"METR says real-world ML research often has slower feedback and less clearly defined goals, leaving uncertainty about whether research can be packaged into small tasks for models to iterate on.","excerpt":"However, real-world ML R&D work often has much slower feedback loops and less clearly-defined goals than in our environments, and it’s unclear to what extent research problems can be packaged into small tasks that models can iterate on effectively.","support":"source-stated"},{"text":"METR presents RE-Bench as a possible tool for evaluations of dangerous levels of autonomous AI R&D capabilities.","excerpt":"We hope RE-Bench and the methodology we’ve developed will be useful in creating evaluations for dangerous levels of autonomous AI R&D capabilities.","support":"source-stated"}],"limitations":["Text as retrieved on October 23, 2026; originally published November 22, 2024.","METR notes that the benchmark has a small number of environments and that its clear goals and faster feedback differ from real-world ML research.","METR says better scaffolding and orchestration may improve agent performance; the cited results do not establish how much improvement would follow.","The results concern benchmark tasks and do not establish agents’ performance across real-world ML R&D."],"mediaId":null,"correctionReason":null,"id":"2040a9becbc0291768af785ba162ee97d9336b2038118269de831efabfc4c35f","publicationId":"156ced09-5bc4-3d9d-3e04-00f1d44c1906","version":1,"sourceUrl":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","sourceTitle":"Evaluating frontier AI R&D capabilities of language model agents against human experts","publisher":"METR","publishedAt":"2024-11-22T08:00:00.000Z","originalPublished":{"value":"2024-11-22T08:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T03:43:19.850Z","createdAt":"2026-11-16T10:58:22.973Z","historical":true,"entityIds":["metr"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"049b017a51baa9c662a6db45bb211493e111b53cd1cfa87b9d75db664a85b03e","rubricVersion":"unrecorded","provider":"vercel","model":"typesafe-ai/jev","reasonCodes":["bridge_admitted_by_model"],"confidence":0.6,"decidedAt":"2026-10-23T23:57:42.096Z"},"provenance":{"textHash":"e4c59bfca9d4ec2749a352d2bb84f65412b0c263b44a2997d62b59ce1d4aa742","observationId":"81b12e818d59f4eeea6ab08acd215d5a54c85d83be5bfd727a232bc3b026b71a","retrievedAt":"2026-10-23T23:59:37.953Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"8a8d6cd5-80bb-1421-321d-486a9f03dcb0"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"dc459273-22b8-f816-55e1-4c4aabe3dbcf"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-16T10:53:43.858Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"156ced09-5bc4-3d9d-3e04-00f1d44c1906","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading is a single METR blog post announcing RE-Bench, and the reading attributes it to METR. Results and hopes are kept as METR's reported findings and statements.","excerpt":"We’re releasing RE-Bench, a new benchmark for measuring the performance of humans and frontier model agents on ML research engineering tasks."},{"dimension":"summary","status":"uncertain","explanation":"The summary is faithful and preserves METR's main tension: agents win at 2 hours, humans win at longer budgets, and real-world transfer is uncertain. But 'scaffolding and orchestration' carries the cost-performance argument and is undefined. The source glosses scaffolds as setups that let a model repeatedly run code. Claude 3.5 Sonnet and o1-preview are also not identified as frontier models from Anthropic and OpenAI. Smallest repair: add a term-library entry for scaffolding and a short clause naming the models' developers.","excerpt":"Modular is a very simple baseline scaffolding that just lets the model repeatedly run code and see the results"},{"dimension":"date","status":"pass","explanation":"November 22, 2024 matches the UTC calendar day of evidence.originalPublished (2024-11-22T08:00Z) and the URL path. The kind is publisher-page, so no archive disclosure is needed. There is no hindsight framing. METR's expectation of better elicitation and its plans for more environments are reported as hopes or suggestions, not outcomes.","excerpt":null},{"dimension":"support","status":"pass","explanation":"Each claim is quoted closely and kept attributed: the short-budget agent advantage, the near-2x human lead at 32 hours, the cheaper hourly cost, the struggles with novel information, the real-world gaps and the dangerous-capability evaluation aim. The limitations correctly say the scaffolding gains are not established. The question follows directly from METR's own stated uncertainty.","excerpt":"it’s unclear to what extent research problems can be packaged into small tasks that models can iterate on effectively"},{"dimension":"media","status":"pass","explanation":"No media was selected, and no verified candidates were supplied. This is consistent with the decision record.","excerpt":null}],"createdAt":"2026-11-16T10:53:43.858Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}