{"id":"6f661ea87a189766f17aa71c6931a83bac21b57cd0b75a043a1c07f29abc2de9","title":"METR notes compare ways to measure agent ability","argument":"METR’s notes, originally published on July 24, 2026, review ways to measure an agent’s ability by comparing its score with the resources spent to achieve it. The author argues that no single metric suits every situation: a score at a fixed expenditure can work when performance levels off, but can mislead when more expenditure continues to improve results. Other measures track the expenditure needed for a target score, returns to expenditure, or performance relative to human effort. The choice matters because different measures can describe different aspects of capability. Human-grounded measures can be easier to interpret and, if an agent’s human-equivalent time horizon stays stable across tasks, could help predict performance on a new task without testing it first. But the notes also describe cases where these measures become undefined or uninformative, such as when agents dominate humans across expenditure levels. The author does not recommend a best metric for each situation and does not deeply address the practical difficulties of measuring score curves. The notes conclude with a key qualification for reliability: probability of success alone can be insufficient when agents may abstain, so evaluation must capture both correctness and coverage. The notes leave unclear how much measured performance changes when the expected budget or scoring rule changes.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 24, 2026; originally published July 24, 2026. The notes say they are unpolished research updates that have gone through less review than METR’s research posts and do not necessarily reflect METR’s views as a whole. The author generally does not recommend which metric is best for each situation, deeply discuss how to measure score curves, or assess metrics’ other desirable properties. The author says that the effect of budget and scoring-rule expectations on measured performance is unclear. The notes do not establish how large that effect is in practice.","speaker":{"name":"METR","roleAtSource":"Publisher"},"source":{"title":"Metrics of Agent Ability","publisher":"METR","format":"Article","url":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/","publishedAt":"2026-07-24T07:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"METR’s notes review alternative metrics for agent capability based on agent and human score functions over expenditure.","excerpt":"This post reviews alternative metrics of agent capability.","url":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/","relationship":"Source-stated"},{"text":"The author says fixed-expenditure scores are less useful when agent scores keep increasing at high expenditure.","excerpt":"This metric becomes less useful when agent scores continue increasing even at high levels of expenditure","url":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/","relationship":"Source-stated"},{"text":"The author says human-grounded metrics can help generalize to new tasks if an agent’s human-equivalent time horizon is stable across tasks.","excerpt":"If an agent’s human-equivalent time horizon were stable across tasks, it would allow us to predict its performance on a new task without first testing the agent on that task.","url":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/","relationship":"Source-stated"},{"text":"The author says evaluation of agents that may abstain must capture both correctness and coverage.","excerpt":"If agents may abstain, probability of success among attempted answers is also insufficient: evaluation must jointly capture correctness and coverage.","url":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/","relationship":"Source-stated"},{"text":"The author notes that measurement can be affected by the budget and scoring rule given to humans and agents, and says the practical size of this effect is unclear.","excerpt":"Ideally, both humans and agents should be told their expenditure budget and their scoring rule. It is unclear how big this effect is in practice.","url":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How should evaluators measure score curves when the strategy itself changes with the expected budget or scoring rule?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"METR notes compare ways to measure agent ability","summary":"METR’s notes, originally published on July 24, 2026, review ways to measure an agent’s ability by comparing its score with the resources spent to achieve it. The author argues that no single metric suits every situation: a score at a fixed expenditure can work when performance levels off, but can mislead when more expenditure continues to improve results. Other measures track the expenditure needed for a target score, returns to expenditure, or performance relative to human effort. The choice matters because different measures can describe different aspects of capability. Human-grounded measures can be easier to interpret and, if an agent’s human-equivalent time horizon stays stable across tasks, could help predict performance on a new task without testing it first. But the notes also describe cases where these measures become undefined or uninformative, such as when agents dominate humans across expenditure levels. The author does not recommend a best metric for each situation and does not deeply address the practical difficulties of measuring score curves. The notes conclude with a key qualification for reliability: probability of success alone can be insufficient when agents may abstain, so evaluation must capture both correctness and coverage. The notes leave unclear how much measured performance changes when the expected budget or scoring rule changes.","question":"How should evaluators measure score curves when the strategy itself changes with the expected budget or scoring rule?","claims":[{"text":"METR’s notes review alternative metrics for agent capability based on agent and human score functions over expenditure.","excerpt":"This post reviews alternative metrics of agent capability.","support":"source-stated"},{"text":"The author says fixed-expenditure scores are less useful when agent scores keep increasing at high expenditure.","excerpt":"This metric becomes less useful when agent scores continue increasing even at high levels of expenditure","support":"source-stated"},{"text":"The author says human-grounded metrics can help generalize to new tasks if an agent’s human-equivalent time horizon is stable across tasks.","excerpt":"If an agent’s human-equivalent time horizon were stable across tasks, it would allow us to predict its performance on a new task without first testing the agent on that task.","support":"source-stated"},{"text":"The author says evaluation of agents that may abstain must capture both correctness and coverage.","excerpt":"If agents may abstain, probability of success among attempted answers is also insufficient: evaluation must jointly capture correctness and coverage.","support":"source-stated"},{"text":"The author notes that measurement can be affected by the budget and scoring rule given to humans and agents, and says the practical size of this effect is unclear.","excerpt":"Ideally, both humans and agents should be told their expenditure budget and their scoring rule. It is unclear how big this effect is in practice.","support":"source-stated"}],"limitations":["Text as retrieved on October 24, 2026; originally published July 24, 2026.","The notes say they are unpolished research updates that have gone through less review than METR’s research posts and do not necessarily reflect METR’s views as a whole.","The author generally does not recommend which metric is best for each situation, deeply discuss how to measure score curves, or assess metrics’ other desirable properties.","The author says that the effect of budget and scoring-rule expectations on measured performance is unclear. The notes do not establish how large that effect is in practice."],"mediaId":null,"correctionReason":null,"id":"6f661ea87a189766f17aa71c6931a83bac21b57cd0b75a043a1c07f29abc2de9","publicationId":"8d645327-5640-1c8f-057e-65b237575b63","version":1,"sourceUrl":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/","sourceTitle":"Metrics of Agent Ability","publisher":"METR","publishedAt":"2026-07-24T07:00:00.000Z","originalPublished":{"value":"2026-07-24T07:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T02:47:39.423Z","createdAt":"2026-11-06T07:38:25.980Z","historical":true,"entityIds":["metr"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"29c4dcf1a26e81754c5f73b1e20de0687c151d0d9de97189892e8d58f342d057","rubricVersion":"unrecorded","provider":"unrecorded","model":"anthropic/claude-opus-5.5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-24T03:49:54.277Z"},"provenance":{"textHash":"665129b62a2218cc419971252c25acf5522620bcfd4c0e0606030d8d32a9a7eb","observationId":"012dedcdcb5029eef55a63f3016cfcdf00914cd27395bb75661efecf4b24358e","retrievedAt":"2026-10-24T03:51:37.001Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"e86df649-1dc6-7b38-632f-fd9b49ef8a8a"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"d433634b-d7e3-d2fc-0715-d251dc507a58"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-06T07:34:05.451Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/metr-notes-compare-ways-to-measure-agent-ability-6f661ea8","formats":{"html":"/readings/metr-notes-compare-ways-to-measure-agent-ability-6f661ea8","markdown":"/readings/metr-notes-compare-ways-to-measure-agent-ability-6f661ea8.md","json":"/readings/metr-notes-compare-ways-to-measure-agent-ability-6f661ea8.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-time-horizon","name":"Time horizon","explanation":"METR's measure of AI agents' ability: the length of task, timed by how long skilled people take, that an agent completes at a set success rate.","referenceUrl":"https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/"}],"entities":[{"id":"metr","name":"METR","kind":"organization","context":"METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.","referenceUrls":["https://metr.org/"],"path":"/entities/metr","json":"/entities/metr.json","markdown":"/entities/metr.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"6f661ea87a189766f17aa71c6931a83bac21b57cd0b75a043a1c07f29abc2de9","revision":"8d645327-5640-1c8f-057e-65b237575b63","revisionKind":"publication-id","title":"METR notes compare ways to measure agent ability","url":"https://escapemeridian.com/readings/metr-notes-compare-ways-to-measure-agent-ability-6f661ea8"},"source":{"type":"source","id":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/","revision":"665129b62a2218cc419971252c25acf5522620bcfd4c0e0606030d8d32a9a7eb","revisionKind":"sha256-source-text","title":"Metrics of Agent Ability","url":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"6f661ea87a189766f17aa71c6931a83bac21b57cd0b75a043a1c07f29abc2de9","title":"METR notes compare ways to measure agent ability","argument":"METR’s notes, originally published on July 24, 2026, review ways to measure an agent’s ability by comparing its score with the resources spent to achieve it. The author argues that no single metric suits every situation: a score at a fixed expenditure can work when performance levels off, but can mislead when more expenditure continues to improve results. Other measures track the expenditure needed for a target score, returns to expenditure, or performance relative to human effort. The choice matters because different measures can describe different aspects of capability. Human-grounded measures can be easier to interpret and, if an agent’s human-equivalent time horizon stays stable across tasks, could help predict performance on a new task without testing it first. But the notes also describe cases where these measures become undefined or uninformative, such as when agents dominate humans across expenditure levels. The author does not recommend a best metric for each situation and does not deeply address the practical difficulties of measuring score curves. The notes conclude with a key qualification for reliability: probability of success alone can be insufficient when agents may abstain, so evaluation must capture both correctness and coverage. The notes leave unclear how much measured performance changes when the expected budget or scoring rule changes.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 24, 2026; originally published July 24, 2026. The notes say they are unpolished research updates that have gone through less review than METR’s research posts and do not necessarily reflect METR’s views as a whole. The author generally does not recommend which metric is best for each situation, deeply discuss how to measure score curves, or assess metrics’ other desirable properties. The author says that the effect of budget and scoring-rule expectations on measured performance is unclear. The notes do not establish how large that effect is in practice.","speaker":{"name":"METR","roleAtSource":"Publisher"},"source":{"title":"Metrics of Agent Ability","publisher":"METR","format":"Article","url":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/","publishedAt":"2026-07-24T07:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"METR’s notes review alternative metrics for agent capability based on agent and human score functions over expenditure.","excerpt":"This post reviews alternative metrics of agent capability.","url":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/","relationship":"Source-stated"},{"text":"The author says fixed-expenditure scores are less useful when agent scores keep increasing at high expenditure.","excerpt":"This metric becomes less useful when agent scores continue increasing even at high levels of expenditure","url":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/","relationship":"Source-stated"},{"text":"The author says human-grounded metrics can help generalize to new tasks if an agent’s human-equivalent time horizon is stable across tasks.","excerpt":"If an agent’s human-equivalent time horizon were stable across tasks, it would allow us to predict its performance on a new task without first testing the agent on that task.","url":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/","relationship":"Source-stated"},{"text":"The author says evaluation of agents that may abstain must capture both correctness and coverage.","excerpt":"If agents may abstain, probability of success among attempted answers is also insufficient: evaluation must jointly capture correctness and coverage.","url":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/","relationship":"Source-stated"},{"text":"The author notes that measurement can be affected by the budget and scoring rule given to humans and agents, and says the practical size of this effect is unclear.","excerpt":"Ideally, both humans and agents should be told their expenditure budget and their scoring rule. It is unclear how big this effect is in practice.","url":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How should evaluators measure score curves when the strategy itself changes with the expected budget or scoring rule?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"METR notes compare ways to measure agent ability","summary":"METR’s notes, originally published on July 24, 2026, review ways to measure an agent’s ability by comparing its score with the resources spent to achieve it. The author argues that no single metric suits every situation: a score at a fixed expenditure can work when performance levels off, but can mislead when more expenditure continues to improve results. Other measures track the expenditure needed for a target score, returns to expenditure, or performance relative to human effort. The choice matters because different measures can describe different aspects of capability. Human-grounded measures can be easier to interpret and, if an agent’s human-equivalent time horizon stays stable across tasks, could help predict performance on a new task without testing it first. But the notes also describe cases where these measures become undefined or uninformative, such as when agents dominate humans across expenditure levels. The author does not recommend a best metric for each situation and does not deeply address the practical difficulties of measuring score curves. The notes conclude with a key qualification for reliability: probability of success alone can be insufficient when agents may abstain, so evaluation must capture both correctness and coverage. The notes leave unclear how much measured performance changes when the expected budget or scoring rule changes.","question":"How should evaluators measure score curves when the strategy itself changes with the expected budget or scoring rule?","claims":[{"text":"METR’s notes review alternative metrics for agent capability based on agent and human score functions over expenditure.","excerpt":"This post reviews alternative metrics of agent capability.","support":"source-stated"},{"text":"The author says fixed-expenditure scores are less useful when agent scores keep increasing at high expenditure.","excerpt":"This metric becomes less useful when agent scores continue increasing even at high levels of expenditure","support":"source-stated"},{"text":"The author says human-grounded metrics can help generalize to new tasks if an agent’s human-equivalent time horizon is stable across tasks.","excerpt":"If an agent’s human-equivalent time horizon were stable across tasks, it would allow us to predict its performance on a new task without first testing the agent on that task.","support":"source-stated"},{"text":"The author says evaluation of agents that may abstain must capture both correctness and coverage.","excerpt":"If agents may abstain, probability of success among attempted answers is also insufficient: evaluation must jointly capture correctness and coverage.","support":"source-stated"},{"text":"The author notes that measurement can be affected by the budget and scoring rule given to humans and agents, and says the practical size of this effect is unclear.","excerpt":"Ideally, both humans and agents should be told their expenditure budget and their scoring rule. It is unclear how big this effect is in practice.","support":"source-stated"}],"limitations":["Text as retrieved on October 24, 2026; originally published July 24, 2026.","The notes say they are unpolished research updates that have gone through less review than METR’s research posts and do not necessarily reflect METR’s views as a whole.","The author generally does not recommend which metric is best for each situation, deeply discuss how to measure score curves, or assess metrics’ other desirable properties.","The author says that the effect of budget and scoring-rule expectations on measured performance is unclear. The notes do not establish how large that effect is in practice."],"mediaId":null,"correctionReason":null,"id":"6f661ea87a189766f17aa71c6931a83bac21b57cd0b75a043a1c07f29abc2de9","publicationId":"8d645327-5640-1c8f-057e-65b237575b63","version":1,"sourceUrl":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/","sourceTitle":"Metrics of Agent Ability","publisher":"METR","publishedAt":"2026-07-24T07:00:00.000Z","originalPublished":{"value":"2026-07-24T07:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T02:47:39.423Z","createdAt":"2026-11-06T07:38:25.980Z","historical":true,"entityIds":["metr"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/notes/2026-07-24-metrics-of-model-ability/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"29c4dcf1a26e81754c5f73b1e20de0687c151d0d9de97189892e8d58f342d057","rubricVersion":"unrecorded","provider":"unrecorded","model":"anthropic/claude-opus-5.5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-24T03:49:54.277Z"},"provenance":{"textHash":"665129b62a2218cc419971252c25acf5522620bcfd4c0e0606030d8d32a9a7eb","observationId":"012dedcdcb5029eef55a63f3016cfcdf00914cd27395bb75661efecf4b24358e","retrievedAt":"2026-10-24T03:51:37.001Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"e86df649-1dc6-7b38-632f-fd9b49ef8a8a"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"d433634b-d7e3-d2fc-0715-d251dc507a58"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-06T07:34:05.451Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"8d645327-5640-1c8f-057e-65b237575b63","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the piece to a METR note by an unnamed author. It keeps the source's caveat that notes are less reviewed and do not necessarily reflect METR's views as a whole. This is a single argued piece, not a listing.","excerpt":"They do not necessarily reflect the views of METR as a whole."},{"dimension":"summary","status":"uncertain","explanation":"Two passages need repair. 1) The summary says the notes 'conclude with' the abstention and coverage point. That point appears mid-FAQ; the notes actually end with human-agent hybrid scoring and related papers. Smallest repair: replace 'conclude with' with 'also include'. 2) 'The author argues that no single metric suits every situation' overstates the source. The author explicitly declines to recommend metrics, while noting conditions under which some metrics work. Smallest repair: 'the author describes when different metrics work and does not recommend one.' Otherwise the summary is readable, and the time horizon definition covers that term.","excerpt":"I will generally not give recommendations of which metric is the best for which situation."},{"dimension":"date","status":"pass","explanation":"The reading gives July 24, 2026, which matches the UTC day of evidence.originalPublished and the URL path. It relies on no later events and frames no forecasts as outcomes. The evidence is a publisher page, so no archive disclosure is required.","excerpt":null},{"dimension":"support","status":"pass","explanation":"Each claim is directly stated in the source with an exact excerpt. The point about interpretability is supported by the source's statement on the advantages of human-grounded metrics. The conditional on prediction from a stable time horizon is preserved. The closing question follows from the source's stated uncertainty about how budget and scoring-rule expectations affect measurement.","excerpt":"An advantage of human-grounded metrics is their interpretability and generalizability."},{"dimension":"media","status":"pass","explanation":"No media was selected and no verified candidates were supplied, so nothing is misidentified.","excerpt":null}],"createdAt":"2026-11-06T07:34:05.451Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}