# METR outlines a bounty for hard, varied tasks for autonomous LLM agents

[Canonical reading](https://escapemeridian.com/readings/metr-outlines-a-bounty-for-hard-varied-tasks-for-autonomous-llm-3c18ab27)

METR’s post, published December 16, 2023, seeks ideas, specifications, and implementations of tasks to measure autonomous LLM agents. It calls for tasks that are hard for human professionals, varied in their bottlenecks, and designed so success or failure reflects ability rather than bugs, shortcuts, or setup quirks. METR argues that measuring a broad range of capabilities matters because ruling out specific threat scenarios may not address the danger from an agent able to carry out varied, expert-level projects. It wants tasks spanning areas such as software engineering, cybersecurity, research, and AI R&D, with a fairly uniform range of difficulty so gradual changes in performance can be interpreted. The post also stresses independent quality assurance to catch ambiguous instructions, missing resources, and scoring or environment problems that could distort results. Its proposed measure is intended to cover an agent’s ability to do things in the world and succeed at tasks, including work relevant to threat models. The post does not report how well tasks built to these criteria measure agent abilities in practice.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

METR · Article

Original source date: Dec 16, 2023 (2023-12-16T20:00:00.000Z).

Date basis: Date supplied by the publisher’s feed.

Added to Escape Meridian: 2026-11-22T04:18:13.359Z. First observed: 2026-10-18T03:57:19.733Z. Retrieved: 2026-10-25T03:43:45.098Z.

- [Bounty: Diverse hard tasks for LLM agents](https://metr.org/blog/2023-12-16-bounty-diverse-hard-tasks-for-llm-agents/)

## Limits

Text as retrieved on October 25, 2026; originally published December 16, 2023. The post does not report results showing how well tasks built to these criteria measure agent abilities in practice. The retrieved page includes a March 14, 2024 update and a February 5, 2024 payment update, both later than the original publication date; they are not treated here as part of the original post.


## Open question

How well do tasks built to these criteria distinguish an agent’s underlying ability from effects of setup, scoring, or task design?


## Source-stated

METR sought ideas, detailed specifications, and tested implementations of tasks for measuring autonomous LLM agents.

> METR (formerly ARC Evals) is looking for (1) ideas, (2) detailed specifications, and (3) well-tested implementations for tasks to measure performance of autonomous LLM agents.

- [Supporting source](https://metr.org/blog/2023-12-16-bounty-diverse-hard-tasks-for-llm-agents/)

## Source-stated

METR argued that broad capability tasks matter because ruling out specific threat scenarios may not address danger from agents able to carry out varied expert-level projects.

> it seems hard to rule out danger from an agent that’s able to carry out a wide variety of multi-day expert-level projects in software development, ML, computer security, research, business, etc. just by ruling out specific threat stories.

- [Supporting source](https://metr.org/blog/2023-12-16-bounty-diverse-hard-tasks-for-llm-agents/)

## Source-stated

METR wanted varied task difficulty and bottlenecks so that task success could provide a more continuous measure of capability.

> We also want the distribution of task difficulty to be fairly uniform, such that we can meaningfully extrapolate gradual increases in task success

- [Supporting source](https://metr.org/blog/2023-12-16-bounty-diverse-hard-tasks-for-llm-agents/)

## Source-stated

METR emphasized independent quality assurance to catch problems that could make an agent’s performance misleading.

> you need to have somebody else run through the task to do quality assurance - making sure that the instructions aren’t ambiguous, all the necessary resources are actually present, there isn’t an accidental shortcut, etc.

- [Supporting source](https://metr.org/blog/2023-12-16-bounty-diverse-hard-tasks-for-llm-agents/)

## Source-stated

The post’s opening update says it is out of date and directs readers to a task development guide.

> Update 3/14/2024: This post is out of date. For current information on the task bounty, see our Task Development Guide.

- [Supporting source](https://metr.org/blog/2023-12-16-bounty-diverse-hard-tasks-for-llm-agents/)

## Source-stated

The post includes a February 5, 2024 update to implementation bounty payments.

> Update from 2/5/2024: For tasks that meet the core desiderata we’ll pay a base price of 3x the estimated $ cost of getting a human to complete the task

- [Supporting source](https://metr.org/blog/2023-12-16-bounty-diverse-hard-tasks-for-llm-agents/)

## Source and contributor context

METR · Publisher

The publisher is identified here. No individual author has been inferred.


## Source media

Media retrieval status: unchecked.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## METR

METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.

- [Explore this context](/entities/metr)
- [Reference](https://metr.org/)

## The World

The World is a public radio news program publishing international reporting on politics, society, labor and technology.

- [Explore this context](/entities/the-world)
- [Reference](https://theworld.org/)

## Editorial provenance

Version 1. Publication ID: dc94e928-0554-eb1b-21c5-1b0cd9a20dd3.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-22T04:14:18.716Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge\_admitted\_by\_model, second\_opinion\_categorical.

Observation: 351df1c1cf468a4060ddd1bc2ae571eed224b980103d1dc90ce901efb03105be. Source text hash: a9da12a189e9a3cffc254f96296f2d738cedba8ea67068b957aa42dcf81afac1.


## Revision and source-check history

### Version 1 — 2026-11-22T04:18:13.359Z

METR’s post, published December 16, 2023, seeks ideas, specifications, and implementations of tasks to measure autonomous LLM agents. It calls for tasks that are hard for human professionals, varied in their bottlenecks, and designed so success or failure reflects ability rather than bugs, shortcuts, or setup quirks. METR argues that measuring a broad range of capabilities matters because ruling out specific threat scenarios may not address the danger from an agent able to carry out varied, expert-level projects. It wants tasks spanning areas such as software engineering, cybersecurity, research, and AI R&D, with a fairly uniform range of difficulty so gradual changes in performance can be interpreted. The post also stresses independent quality assurance to catch ambiguous instructions, missing resources, and scoring or environment problems that could distort results. Its proposed measure is intended to cover an agent’s ability to do things in the world and succeed at tasks, including work relevant to threat models. The post does not report how well tasks built to these criteria measure agent abilities in practice.

## Original source

METR · Article

Original source date: Dec 16, 2023 (2023-12-16T20:00:00.000Z).

Date basis: Date supplied by the publisher’s feed.

Added to Escape Meridian: 2026-11-22T04:18:13.359Z. First observed: 2026-10-18T03:57:19.733Z. Retrieved: 2026-10-25T03:43:45.098Z.

- [Bounty: Diverse hard tasks for LLM agents](https://metr.org/blog/2023-12-16-bounty-diverse-hard-tasks-for-llm-agents/)

## Limits

Text as retrieved on October 25, 2026; originally published December 16, 2023. The post does not report results showing how well tasks built to these criteria measure agent abilities in practice. The retrieved page includes a March 14, 2024 update and a February 5, 2024 payment update, both later than the original publication date; they are not treated here as part of the original post.


## Open question

How well do tasks built to these criteria distinguish an agent’s underlying ability from effects of setup, scoring, or task design?


## Source-stated

METR sought ideas, detailed specifications, and tested implementations of tasks for measuring autonomous LLM agents.

> METR (formerly ARC Evals) is looking for (1) ideas, (2) detailed specifications, and (3) well-tested implementations for tasks to measure performance of autonomous LLM agents.

- [Supporting source](https://metr.org/blog/2023-12-16-bounty-diverse-hard-tasks-for-llm-agents/)

## Source-stated

METR argued that broad capability tasks matter because ruling out specific threat scenarios may not address danger from agents able to carry out varied expert-level projects.

> it seems hard to rule out danger from an agent that’s able to carry out a wide variety of multi-day expert-level projects in software development, ML, computer security, research, business, etc. just by ruling out specific threat stories.

- [Supporting source](https://metr.org/blog/2023-12-16-bounty-diverse-hard-tasks-for-llm-agents/)

## Source-stated

METR wanted varied task difficulty and bottlenecks so that task success could provide a more continuous measure of capability.

> We also want the distribution of task difficulty to be fairly uniform, such that we can meaningfully extrapolate gradual increases in task success

- [Supporting source](https://metr.org/blog/2023-12-16-bounty-diverse-hard-tasks-for-llm-agents/)

## Source-stated

METR emphasized independent quality assurance to catch problems that could make an agent’s performance misleading.

> you need to have somebody else run through the task to do quality assurance - making sure that the instructions aren’t ambiguous, all the necessary resources are actually present, there isn’t an accidental shortcut, etc.

- [Supporting source](https://metr.org/blog/2023-12-16-bounty-diverse-hard-tasks-for-llm-agents/)

## Source-stated

The post’s opening update says it is out of date and directs readers to a task development guide.

> Update 3/14/2024: This post is out of date. For current information on the task bounty, see our Task Development Guide.

- [Supporting source](https://metr.org/blog/2023-12-16-bounty-diverse-hard-tasks-for-llm-agents/)

## Source-stated

The post includes a February 5, 2024 update to implementation bounty payments.

> Update from 2/5/2024: For tasks that meet the core desiderata we’ll pay a base price of 3x the estimated $ cost of getting a human to complete the task

- [Supporting source](https://metr.org/blog/2023-12-16-bounty-diverse-hard-tasks-for-llm-agents/)

## Source and contributor context

METR · Publisher

The publisher is identified here. No individual author has been inferred.


## Source media

Media retrieval status: unchecked.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## METR

METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.

- [Explore this context](/entities/metr)
- [Reference](https://metr.org/)

## The World

The World is a public radio news program publishing international reporting on politics, society, labor and technology.

- [Explore this context](/entities/the-world)
- [Reference](https://theworld.org/)

## Editorial provenance

Version 1. Publication ID: dc94e928-0554-eb1b-21c5-1b0cd9a20dd3.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-22T04:14:18.716Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge\_admitted\_by\_model, second\_opinion\_categorical.

Observation: 351df1c1cf468a4060ddd1bc2ae571eed224b980103d1dc90ce901efb03105be. Source text hash: a9da12a189e9a3cffc254f96296f2d738cedba8ea67068b957aa42dcf81afac1.


### uncertain — 2026-11-22T04:14:18.716Z — anthropic/claude-opus-5.5

Publication: dc94e928-0554-eb1b-21c5-1b0cd9a20dd3

attribution — pass: This is a single METR blog post with its own argument and requirements, not a listing. The reading attributes the bounty, its rationale and its desiderata to METR throughout, and does not present METR's risk reasoning as established fact.

> METR (formerly ARC Evals) is looking for (1) ideas, (2) detailed specifications, and (3) well-tested implementations for tasks to measure performance of autonomous LLM agents.

summary — pass: The reading names the actor (METR) and says what it sought and why. It covers the key criteria: difficulty, varied bottlenecks, uniform difficulty and quality assurance. It conveys METR's argument that a general capability measure matters because ruling out specific threat stories is hard. LLM is defined on the page, and 'agent' and 'threat scenarios' are understandable in context. The question follows from the evidence without leading. The post's framing in terms of catastrophic risk is softened to 'danger', but the core point is preserved.

> an agent that succeeds at most of the tasks seems really quite likely to pose a catastrophic risk

date — uncertain: The stated date, December 16, 2023, matches originalPublished and the URL. However, two claims report the 2/5/2024 payment update and the 3/14/2024 out-of-date notice. Both postdate the original publication, yet the reading is meant to describe the source as of that date. The limitations label these as later updates, so neither is passed off as original. Still, the reading uses post-publication content as claims. The page was also evidently revised afterwards, including a starter pack marked 'now deprecated', so some undated body text may postdate December 2023. The smallest repair is to drop claims 5 and 6, or keep them only as a limitation note.

> Update 3/14/2024: This post is out of date. For current information on the task bounty, see our Task Development Guide.

support — pass: Each claim's excerpt appears verbatim in the source and supports the paraphrase. The statement that the post reports no results on how well the tasks measure ability is accurate. No benefits, costs or outcomes are invented, and the measurement goals are kept as METR's intentions rather than demonstrated results.

> We want tasks that, taken together, allows us to get a fairly continuous measure of “ability to do things in the world and succeed at tasks”.

media — pass: No media was selected, and no verified source media candidates were supplied, so leaving media empty is appropriate.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/3c18ab270df4bdbcdb8f77507b0ee4e1bd4a0deec073e805a415ff8c1325c59f/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/metr-outlines-a-bounty-for-hard-varied-tasks-for-autonomous-llm-3c18ab27.json)
