{"id":"0f04ecafd40829ca4ab46b4ba38bb62a7db314d18b89b4c9deac215d99dbf9c3","title":"METR outlines how to elicit and evaluate model capabilities","argument":"METR published these guidelines on March 15, 2024. They argue that evaluations should measure capabilities a model could reach with plausible post-training enhancement, rather than testing only the model as-is. Evaluators should first try basic elicitation, then sort remaining failures into “spurious” bottlenecks, “real” bottlenecks and “tradeoffs,” because each calls for different handling. The guidelines also recommend checking for overfitting, evaluation gaming and task-level problems, and reporting failures and red flags. METR’s stated concern is that a test-set score can misrepresent model capabilities if evaluators overlook fixable failures or count questionable successes. The proposed process aims to make evaluations harder to game while keeping unresolved failures visible. METR says its recommendations draw on qualitative experience, not a full evaluation using this methodology or experimental confirmation that the failure categories behave as expected. It describes them as informed, reasoned guesses. Whether the categories reliably distinguish failures that can be fixed across tasks from those that cannot remains untested in the evidence presented.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 25, 2026; originally published March 15, 2024. METR says it has not conducted a full evaluation using this methodology or experimentally confirmed the proposed failure categories; the guidelines are based on qualitative experience.","speaker":{"name":"METR","roleAtSource":"Publisher"},"source":{"title":"Guidelines for capability elicitation","publisher":"METR","format":"Article","url":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","publishedAt":"2024-03-15T09:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"METR says the guidelines aim to measure capabilities accessible after plausible post-training enhancement, not only a model as-is.","excerpt":"The goal of this process is to get a test-set score that represents the full capabilities of the model that are likely to be accessible with plausible amounts of post-training enhancement.","url":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","relationship":"Source-stated"},{"text":"METR recommends classifying remaining failures as spurious bottlenecks, real bottlenecks or tradeoffs, and handling them differently.","excerpt":"The core idea here is to break limitations leading to task failures into 3 categories:","url":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","relationship":"Source-stated"},{"text":"METR says the recommendations are based on qualitative experience and have not yet been tested through a full evaluation or experimentally confirmed.","excerpt":"However, we have not yet conducted a full evaluation according to this methodology, nor have we experimentally confirmed that the different types of failures mentioned are natural clusters or behave in the ways we expect.","url":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","relationship":"Source-stated"},{"text":"METR characterizes the recommendations as informed, reasoned guesses.","excerpt":"Our recommendations here should be considered as somewhat informed and reasoned guesses only.","url":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Would a full evaluation using these guidelines show that the proposed failure categories behave as METR expects?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"METR outlines how to elicit and evaluate model capabilities","summary":"METR published these guidelines on March 15, 2024. They argue that evaluations should measure capabilities a model could reach with plausible post-training enhancement, rather than testing only the model as-is. Evaluators should first try basic elicitation, then sort remaining failures into “spurious” bottlenecks, “real” bottlenecks and “tradeoffs,” because each calls for different handling. The guidelines also recommend checking for overfitting, evaluation gaming and task-level problems, and reporting failures and red flags. METR’s stated concern is that a test-set score can misrepresent model capabilities if evaluators overlook fixable failures or count questionable successes. The proposed process aims to make evaluations harder to game while keeping unresolved failures visible. METR says its recommendations draw on qualitative experience, not a full evaluation using this methodology or experimental confirmation that the failure categories behave as expected. It describes them as informed, reasoned guesses. Whether the categories reliably distinguish failures that can be fixed across tasks from those that cannot remains untested in the evidence presented.","question":"Would a full evaluation using these guidelines show that the proposed failure categories behave as METR expects?","claims":[{"text":"METR says the guidelines aim to measure capabilities accessible after plausible post-training enhancement, not only a model as-is.","excerpt":"The goal of this process is to get a test-set score that represents the full capabilities of the model that are likely to be accessible with plausible amounts of post-training enhancement.","support":"source-stated"},{"text":"METR recommends classifying remaining failures as spurious bottlenecks, real bottlenecks or tradeoffs, and handling them differently.","excerpt":"The core idea here is to break limitations leading to task failures into 3 categories:","support":"source-stated"},{"text":"METR says the recommendations are based on qualitative experience and have not yet been tested through a full evaluation or experimentally confirmed.","excerpt":"However, we have not yet conducted a full evaluation according to this methodology, nor have we experimentally confirmed that the different types of failures mentioned are natural clusters or behave in the ways we expect.","support":"source-stated"},{"text":"METR characterizes the recommendations as informed, reasoned guesses.","excerpt":"Our recommendations here should be considered as somewhat informed and reasoned guesses only.","support":"source-stated"}],"limitations":["Text as retrieved on October 25, 2026; originally published March 15, 2024.","METR says it has not conducted a full evaluation using this methodology or experimentally confirmed the proposed failure categories; the guidelines are based on qualitative experience."],"mediaId":null,"correctionReason":null,"id":"0f04ecafd40829ca4ab46b4ba38bb62a7db314d18b89b4c9deac215d99dbf9c3","publicationId":"e78f8be5-d9a7-f643-132e-388297bdcbb7","version":1,"sourceUrl":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","sourceTitle":"Guidelines for capability elicitation","publisher":"METR","publishedAt":"2024-03-15T09:00:00.000Z","originalPublished":{"value":"2024-03-15T09:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T03:53:25.513Z","createdAt":"2026-11-20T23:13:10.185Z","historical":true,"entityIds":["metr"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"0794648791ff5d4e2cf47a30c879ecc3d375f15df972051398bae9e62332886a","rubricVersion":"unrecorded","provider":"unrecorded","model":"anthropic/claude-opus-5.5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-25T03:13:59.282Z"},"provenance":{"textHash":"16496602fd3d1afefbd9ce31f486e5b31c1ff47778450201afe56cb9fb495e01","observationId":"0b21ec5e4fc7f5797712c62a049871b19f5d8b3fa688bd49824789632cff44b0","retrievedAt":"2026-10-25T03:15:40.643Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"1c6e0d2d-a407-8aad-8fdb-529d6debfb76"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"a2a395e4-08dc-030c-ee0c-1c4ba7ea1b3d"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-20T23:08:34.739Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/metr-outlines-how-to-elicit-and-evaluate-model-capabilities-0f04ecaf","formats":{"html":"/readings/metr-outlines-how-to-elicit-and-evaluate-model-capabilities-0f04ecaf","markdown":"/readings/metr-outlines-how-to-elicit-and-evaluate-model-capabilities-0f04ecaf.md","json":"/readings/metr-outlines-how-to-elicit-and-evaluate-model-capabilities-0f04ecaf.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-overfitting","name":"Overfitting","explanation":"When a model learns its training examples too closely and performs worse on new data.","referenceUrl":"https://developers.google.com/machine-learning/glossary#overfitting"},{"id":"term-held-out-set","name":"Held-out set","explanation":"Data kept aside during training and used only to test how well a model works on examples it has not seen.","referenceUrl":"https://developers.google.com/machine-learning/glossary#holdout-data"}],"entities":[{"id":"metr","name":"METR","kind":"organization","context":"METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.","referenceUrls":["https://metr.org/"],"path":"/entities/metr","json":"/entities/metr.json","markdown":"/entities/metr.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"0f04ecafd40829ca4ab46b4ba38bb62a7db314d18b89b4c9deac215d99dbf9c3","revision":"e78f8be5-d9a7-f643-132e-388297bdcbb7","revisionKind":"publication-id","title":"METR outlines how to elicit and evaluate model capabilities","url":"https://escapemeridian.com/readings/metr-outlines-how-to-elicit-and-evaluate-model-capabilities-0f04ecaf"},"source":{"type":"source","id":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","revision":"16496602fd3d1afefbd9ce31f486e5b31c1ff47778450201afe56cb9fb495e01","revisionKind":"sha256-source-text","title":"Guidelines for capability elicitation","url":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"0f04ecafd40829ca4ab46b4ba38bb62a7db314d18b89b4c9deac215d99dbf9c3","title":"METR outlines how to elicit and evaluate model capabilities","argument":"METR published these guidelines on March 15, 2024. They argue that evaluations should measure capabilities a model could reach with plausible post-training enhancement, rather than testing only the model as-is. Evaluators should first try basic elicitation, then sort remaining failures into “spurious” bottlenecks, “real” bottlenecks and “tradeoffs,” because each calls for different handling. The guidelines also recommend checking for overfitting, evaluation gaming and task-level problems, and reporting failures and red flags. METR’s stated concern is that a test-set score can misrepresent model capabilities if evaluators overlook fixable failures or count questionable successes. The proposed process aims to make evaluations harder to game while keeping unresolved failures visible. METR says its recommendations draw on qualitative experience, not a full evaluation using this methodology or experimental confirmation that the failure categories behave as expected. It describes them as informed, reasoned guesses. Whether the categories reliably distinguish failures that can be fixed across tasks from those that cannot remains untested in the evidence presented.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 25, 2026; originally published March 15, 2024. METR says it has not conducted a full evaluation using this methodology or experimentally confirmed the proposed failure categories; the guidelines are based on qualitative experience.","speaker":{"name":"METR","roleAtSource":"Publisher"},"source":{"title":"Guidelines for capability elicitation","publisher":"METR","format":"Article","url":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","publishedAt":"2024-03-15T09:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"METR says the guidelines aim to measure capabilities accessible after plausible post-training enhancement, not only a model as-is.","excerpt":"The goal of this process is to get a test-set score that represents the full capabilities of the model that are likely to be accessible with plausible amounts of post-training enhancement.","url":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","relationship":"Source-stated"},{"text":"METR recommends classifying remaining failures as spurious bottlenecks, real bottlenecks or tradeoffs, and handling them differently.","excerpt":"The core idea here is to break limitations leading to task failures into 3 categories:","url":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","relationship":"Source-stated"},{"text":"METR says the recommendations are based on qualitative experience and have not yet been tested through a full evaluation or experimentally confirmed.","excerpt":"However, we have not yet conducted a full evaluation according to this methodology, nor have we experimentally confirmed that the different types of failures mentioned are natural clusters or behave in the ways we expect.","url":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","relationship":"Source-stated"},{"text":"METR characterizes the recommendations as informed, reasoned guesses.","excerpt":"Our recommendations here should be considered as somewhat informed and reasoned guesses only.","url":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Would a full evaluation using these guidelines show that the proposed failure categories behave as METR expects?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"METR outlines how to elicit and evaluate model capabilities","summary":"METR published these guidelines on March 15, 2024. They argue that evaluations should measure capabilities a model could reach with plausible post-training enhancement, rather than testing only the model as-is. Evaluators should first try basic elicitation, then sort remaining failures into “spurious” bottlenecks, “real” bottlenecks and “tradeoffs,” because each calls for different handling. The guidelines also recommend checking for overfitting, evaluation gaming and task-level problems, and reporting failures and red flags. METR’s stated concern is that a test-set score can misrepresent model capabilities if evaluators overlook fixable failures or count questionable successes. The proposed process aims to make evaluations harder to game while keeping unresolved failures visible. METR says its recommendations draw on qualitative experience, not a full evaluation using this methodology or experimental confirmation that the failure categories behave as expected. It describes them as informed, reasoned guesses. Whether the categories reliably distinguish failures that can be fixed across tasks from those that cannot remains untested in the evidence presented.","question":"Would a full evaluation using these guidelines show that the proposed failure categories behave as METR expects?","claims":[{"text":"METR says the guidelines aim to measure capabilities accessible after plausible post-training enhancement, not only a model as-is.","excerpt":"The goal of this process is to get a test-set score that represents the full capabilities of the model that are likely to be accessible with plausible amounts of post-training enhancement.","support":"source-stated"},{"text":"METR recommends classifying remaining failures as spurious bottlenecks, real bottlenecks or tradeoffs, and handling them differently.","excerpt":"The core idea here is to break limitations leading to task failures into 3 categories:","support":"source-stated"},{"text":"METR says the recommendations are based on qualitative experience and have not yet been tested through a full evaluation or experimentally confirmed.","excerpt":"However, we have not yet conducted a full evaluation according to this methodology, nor have we experimentally confirmed that the different types of failures mentioned are natural clusters or behave in the ways we expect.","support":"source-stated"},{"text":"METR characterizes the recommendations as informed, reasoned guesses.","excerpt":"Our recommendations here should be considered as somewhat informed and reasoned guesses only.","support":"source-stated"}],"limitations":["Text as retrieved on October 25, 2026; originally published March 15, 2024.","METR says it has not conducted a full evaluation using this methodology or experimentally confirmed the proposed failure categories; the guidelines are based on qualitative experience."],"mediaId":null,"correctionReason":null,"id":"0f04ecafd40829ca4ab46b4ba38bb62a7db314d18b89b4c9deac215d99dbf9c3","publicationId":"e78f8be5-d9a7-f643-132e-388297bdcbb7","version":1,"sourceUrl":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","sourceTitle":"Guidelines for capability elicitation","publisher":"METR","publishedAt":"2024-03-15T09:00:00.000Z","originalPublished":{"value":"2024-03-15T09:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T03:53:25.513Z","createdAt":"2026-11-20T23:13:10.185Z","historical":true,"entityIds":["metr"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"0794648791ff5d4e2cf47a30c879ecc3d375f15df972051398bae9e62332886a","rubricVersion":"unrecorded","provider":"unrecorded","model":"anthropic/claude-opus-5.5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-25T03:13:59.282Z"},"provenance":{"textHash":"16496602fd3d1afefbd9ce31f486e5b31c1ff47778450201afe56cb9fb495e01","observationId":"0b21ec5e4fc7f5797712c62a049871b19f5d8b3fa688bd49824789632cff44b0","retrievedAt":"2026-10-25T03:15:40.643Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"1c6e0d2d-a407-8aad-8fdb-529d6debfb76"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"a2a395e4-08dc-030c-ee0c-1c4ba7ea1b3d"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-20T23:08:34.739Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"e78f8be5-d9a7-f643-132e-388297bdcbb7","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading consistently attributes the guidelines, aims and caveats to METR, the publisher of this single blog post. Source assertions stay framed as METR's recommendations rather than established fact.","excerpt":"This is an example set of guidelines for eliciting models against a test suite"},{"dimension":"summary","status":"uncertain","explanation":"The three failure categories are central, but they are only named. A non-specialist is not told that spurious failures are easy, universal fixes such as bugs or refusals, that real failures have no obvious fix, or that tradeoffs are fixes that help one task but may hurt others. Elicitation and evaluation gaming are also unexplained, and the reading omits METR's stated reason: risk threat models such as open-sourcing, theft or users probing the model. Smallest repair: add one plain clause defining each category and one clause giving that reason.","excerpt":"“Spurious” bottlenecks: failures that are easily fixable and not “real” capability limitations"},{"dimension":"date","status":"pass","explanation":"The reading gives March 15, 2024, which matches the UTC calendar day of originalPublished and the URL. The evidence kind is publisher-page, so no archive disclosure is needed. The reading uses no later material and frames nothing in hindsight.","excerpt":null},{"dimension":"support","status":"pass","explanation":"The claims match source text: the aim of measuring capabilities reachable with plausible enhancement, the three categories, and the stated lack of a full evaluation. The concern about questionable successes reflects METR's section on spurious successes. The gaming point reflects METR's text on elicitation making gaming harder, and keeping failures visible reflects its instruction to leave remaining failures. The question follows from METR's own stated caveat.","excerpt":"Having the evaluation involve active elicitation effort makes that kind of “evaluation gaming” harder."},{"dimension":"media","status":"pass","explanation":"No media was selected and none was supplied, so there is nothing to misidentify.","excerpt":null}],"createdAt":"2026-11-20T23:08:34.739Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}