METR says its developer productivity estimate is a bad proxy

METR says its developer productivity estimate is a bad proxy

METR’s page was published on February 24, 2026. METR says selection effects make its new experiment a poor measure of AI’s productivity impact on participating open-source developers. Developers who expect AI to help them more are less likely to join or to submit tasks they would rather do with AI; METR says this likely makes its estimate a lower bound. A lower pay rate may also have contributed to selection, and time measurements can be unreliable when developers use multiple AI agents concurrently. METR says the data offer only very weak evidence about the size of any increase in productivity, despite raw results showing some evidence for speedup. It calls the experiment’s central estimate likely a bad proxy for the real productivity impact of AI tools on these developers. METR says developer productivity is an important input to AI R&D acceleration and plans to explore other approaches, including shorter, more intensive experiments, observational data, questionnaires, fixed-task experiments, evaluations and developer-level experiments. Those approaches have limitations, and METR expects to keep redesigning parts of its measurement techniques as model capabilities increase. How much higher pay or shorter experiments would reduce the selection effects remains unknown.

Publication: 0d1ef902-2393-9301-fb20-07f24a4df975

attribution — pass: This is a single METR blog post with its own argument. The reading attributes its claims to METR throughout, and none are presented as established fact or Escape Meridian's view.

we believe that the data from our new experiment gives us an unreliable signal of the current productivity effect of AI tools

summary — uncertain: The reading omits what changed. It never says METR's earlier study found AI caused a roughly 20% slowdown, or that METR now believes developers are likely more sped up in early 2026. Writing 'weak evidence about the size of any increase' turns METR's stated belief in an increase into a vague possibility. Smallest repair: add one sentence on the prior slowdown finding and METR's belief that speedup is now likely larger. Separately, 'Those approaches have limitations' overgeneralizes; the source flags limitations for some approaches, such as questionnaires and developer-level experiments, not all.

we believe it is likely that developers are more sped up from AI tools now — in early 2026 — compared to our estimates from early 2025

date — pass: The reading states February 24, 2026, which matches the UTC day of originalPublished. It uses no later-dated material or hindsight framing. Kind is publisher-page, so no archive disclosure is needed.

support — pass: The claims on non-participation, task selection, the lower-bound estimate, the lower pay rate, unreliable time measurement with concurrent agents, the bad-proxy judgment, and the planned alternative methods are all directly stated in the source. The question follows from METR's note that shorter, more intensive experiments and possibly higher pay could alleviate selection.

The selection issues would be alleviated if we had higher compliance, which could be achieved if we run shorter intense experiments, and possibly pay higher rates.

media — pass: No media was selected, and no verified candidates were supplied. This is appropriate.