METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Loading distillations…
METR reports that selection effects and concurrent agent use make its newer developer experiment unreliable for estimating current speedup. It considers improved productivity plausible while declining to assign a dependable effect size.
Compare benchmark capability with task completion, reliability and actual practitioner workflows.
Selected tasks and participants do not represent every job or deployment; methodology changes matter.
Public studies, methods and linked research artifacts