# METR outlines how to elicit and evaluate model capabilities

[Canonical reading](https://escapemeridian.com/readings/metr-outlines-how-to-elicit-and-evaluate-model-capabilities-0f04ecaf)

METR published these guidelines on March 15, 2024. They argue that evaluations should measure capabilities a model could reach with plausible post-training enhancement, rather than testing only the model as-is. Evaluators should first try basic elicitation, then sort remaining failures into “spurious” bottlenecks, “real” bottlenecks and “tradeoffs,” because each calls for different handling. The guidelines also recommend checking for overfitting, evaluation gaming and task-level problems, and reporting failures and red flags. METR’s stated concern is that a test-set score can misrepresent model capabilities if evaluators overlook fixable failures or count questionable successes. The proposed process aims to make evaluations harder to game while keeping unresolved failures visible. METR says its recommendations draw on qualitative experience, not a full evaluation using this methodology or experimental confirmation that the failure categories behave as expected. It describes them as informed, reasoned guesses. Whether the categories reliably distinguish failures that can be fixed across tasks from those that cannot remains untested in the evidence presented.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

METR · Article

Original source date: Mar 15, 2024 (2024-03-15T09:00:00.000Z).

Date basis: Date supplied by the publisher’s feed.

Added to Escape Meridian: 2026-11-20T23:13:10.185Z. First observed: 2026-10-18T03:53:25.513Z. Retrieved: 2026-10-25T03:15:40.643Z.

- [Guidelines for capability elicitation](https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/)

## Limits

Text as retrieved on October 25, 2026; originally published March 15, 2024. METR says it has not conducted a full evaluation using this methodology or experimentally confirmed the proposed failure categories; the guidelines are based on qualitative experience.


## Open question

Would a full evaluation using these guidelines show that the proposed failure categories behave as METR expects?


## Source-stated

METR says the guidelines aim to measure capabilities accessible after plausible post-training enhancement, not only a model as-is.

> The goal of this process is to get a test-set score that represents the full capabilities of the model that are likely to be accessible with plausible amounts of post-training enhancement.

- [Supporting source](https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/)

## Source-stated

METR recommends classifying remaining failures as spurious bottlenecks, real bottlenecks or tradeoffs, and handling them differently.

> The core idea here is to break limitations leading to task failures into 3 categories:

- [Supporting source](https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/)

## Source-stated

METR says the recommendations are based on qualitative experience and have not yet been tested through a full evaluation or experimentally confirmed.

> However, we have not yet conducted a full evaluation according to this methodology, nor have we experimentally confirmed that the different types of failures mentioned are natural clusters or behave in the ways we expect.

- [Supporting source](https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/)

## Source-stated

METR characterizes the recommendations as informed, reasoned guesses.

> Our recommendations here should be considered as somewhat informed and reasoned guesses only.

- [Supporting source](https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/)

## Source and contributor context

METR · Publisher

The publisher is identified here. No individual author has been inferred.


## Source media

Media retrieval status: unchecked.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Overfitting

When a model learns its training examples too closely and performs worse on new data.

- [Definition reference](https://developers.google.com/machine-learning/glossary#overfitting)

## Held-out set

Data kept aside during training and used only to test how well a model works on examples it has not seen.

- [Definition reference](https://developers.google.com/machine-learning/glossary#holdout-data)

## METR

METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.

- [Explore this context](/entities/metr)
- [Reference](https://metr.org/)

## Editorial provenance

Version 1. Publication ID: e78f8be5-d9a7-f643-132e-388297bdcbb7.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-20T23:08:34.739Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge\_admitted\_by\_model, second\_opinion\_categorical.

Observation: 0b21ec5e4fc7f5797712c62a049871b19f5d8b3fa688bd49824789632cff44b0. Source text hash: 16496602fd3d1afefbd9ce31f486e5b31c1ff47778450201afe56cb9fb495e01.


## Revision and source-check history

### Version 1 — 2026-11-20T23:13:10.185Z

METR published these guidelines on March 15, 2024. They argue that evaluations should measure capabilities a model could reach with plausible post-training enhancement, rather than testing only the model as-is. Evaluators should first try basic elicitation, then sort remaining failures into “spurious” bottlenecks, “real” bottlenecks and “tradeoffs,” because each calls for different handling. The guidelines also recommend checking for overfitting, evaluation gaming and task-level problems, and reporting failures and red flags. METR’s stated concern is that a test-set score can misrepresent model capabilities if evaluators overlook fixable failures or count questionable successes. The proposed process aims to make evaluations harder to game while keeping unresolved failures visible. METR says its recommendations draw on qualitative experience, not a full evaluation using this methodology or experimental confirmation that the failure categories behave as expected. It describes them as informed, reasoned guesses. Whether the categories reliably distinguish failures that can be fixed across tasks from those that cannot remains untested in the evidence presented.

## Original source

METR · Article

Original source date: Mar 15, 2024 (2024-03-15T09:00:00.000Z).

Date basis: Date supplied by the publisher’s feed.

Added to Escape Meridian: 2026-11-20T23:13:10.185Z. First observed: 2026-10-18T03:53:25.513Z. Retrieved: 2026-10-25T03:15:40.643Z.

- [Guidelines for capability elicitation](https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/)

## Limits

Text as retrieved on October 25, 2026; originally published March 15, 2024. METR says it has not conducted a full evaluation using this methodology or experimentally confirmed the proposed failure categories; the guidelines are based on qualitative experience.


## Open question

Would a full evaluation using these guidelines show that the proposed failure categories behave as METR expects?


## Source-stated

METR says the guidelines aim to measure capabilities accessible after plausible post-training enhancement, not only a model as-is.

> The goal of this process is to get a test-set score that represents the full capabilities of the model that are likely to be accessible with plausible amounts of post-training enhancement.

- [Supporting source](https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/)

## Source-stated

METR recommends classifying remaining failures as spurious bottlenecks, real bottlenecks or tradeoffs, and handling them differently.

> The core idea here is to break limitations leading to task failures into 3 categories:

- [Supporting source](https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/)

## Source-stated

METR says the recommendations are based on qualitative experience and have not yet been tested through a full evaluation or experimentally confirmed.

> However, we have not yet conducted a full evaluation according to this methodology, nor have we experimentally confirmed that the different types of failures mentioned are natural clusters or behave in the ways we expect.

- [Supporting source](https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/)

## Source-stated

METR characterizes the recommendations as informed, reasoned guesses.

> Our recommendations here should be considered as somewhat informed and reasoned guesses only.

- [Supporting source](https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/)

## Source and contributor context

METR · Publisher

The publisher is identified here. No individual author has been inferred.


## Source media

Media retrieval status: unchecked.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Overfitting

When a model learns its training examples too closely and performs worse on new data.

- [Definition reference](https://developers.google.com/machine-learning/glossary#overfitting)

## Held-out set

Data kept aside during training and used only to test how well a model works on examples it has not seen.

- [Definition reference](https://developers.google.com/machine-learning/glossary#holdout-data)

## METR

METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.

- [Explore this context](/entities/metr)
- [Reference](https://metr.org/)

## Editorial provenance

Version 1. Publication ID: e78f8be5-d9a7-f643-132e-388297bdcbb7.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-20T23:08:34.739Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge\_admitted\_by\_model, second\_opinion\_categorical.

Observation: 0b21ec5e4fc7f5797712c62a049871b19f5d8b3fa688bd49824789632cff44b0. Source text hash: 16496602fd3d1afefbd9ce31f486e5b31c1ff47778450201afe56cb9fb495e01.


### uncertain — 2026-11-20T23:08:34.739Z — anthropic/claude-opus-5.5

Publication: e78f8be5-d9a7-f643-132e-388297bdcbb7

attribution — pass: The reading consistently attributes the guidelines, aims and caveats to METR, the publisher of this single blog post. Source assertions stay framed as METR's recommendations rather than established fact.

> This is an example set of guidelines for eliciting models against a test suite

summary — uncertain: The three failure categories are central, but they are only named. A non-specialist is not told that spurious failures are easy, universal fixes such as bugs or refusals, that real failures have no obvious fix, or that tradeoffs are fixes that help one task but may hurt others. Elicitation and evaluation gaming are also unexplained, and the reading omits METR's stated reason: risk threat models such as open-sourcing, theft or users probing the model. Smallest repair: add one plain clause defining each category and one clause giving that reason.

> “Spurious” bottlenecks: failures that are easily fixable and not “real” capability limitations

date — pass: The reading gives March 15, 2024, which matches the UTC calendar day of originalPublished and the URL. The evidence kind is publisher-page, so no archive disclosure is needed. The reading uses no later material and frames nothing in hindsight.

support — pass: The claims match source text: the aim of measuring capabilities reachable with plausible enhancement, the three categories, and the stated lack of a full evaluation. The concern about questionable successes reflects METR's section on spurious successes. The gaming point reflects METR's text on elicitation making gaming harder, and keeping failures visible reflects its instruction to leave remaining failures. The question follows from METR's own stated caveat.

> Having the evaluation involve active elicitation effort makes that kind of “evaluation gaming” harder.

media — pass: No media was selected and none was supplied, so there is nothing to misidentify.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/0f04ecafd40829ca4ab46b4ba38bb62a7db314d18b89b4c9deac215d99dbf9c3/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/metr-outlines-how-to-elicit-and-evaluate-model-capabilities-0f04ecaf.json)
