{"id":"54d658ff870e4bbb0638a847bac138a3b543d42ac48e761c915c570500f1cc5c","title":"METR says its developer productivity estimate is a bad proxy","argument":"METR’s page was published on February 24, 2026. METR says selection effects make its new experiment a poor measure of AI’s productivity impact on participating open-source developers. Developers who expect AI to help them more are less likely to join or to submit tasks they would rather do with AI; METR says this likely makes its estimate a lower bound. A lower pay rate may also have contributed to selection, and time measurements can be unreliable when developers use multiple AI agents concurrently. METR says the data offer only very weak evidence about the size of any increase in productivity, despite raw results showing some evidence for speedup. It calls the experiment’s central estimate likely a bad proxy for the real productivity impact of AI tools on these developers. METR says developer productivity is an important input to AI R&D acceleration and plans to explore other approaches, including shorter, more intensive experiments, observational data, questionnaires, fixed-task experiments, evaluations and developer-level experiments. Those approaches have limitations, and METR expects to keep redesigning parts of its measurement techniques as model capabilities increase. How much higher pay or shorter experiments would reduce the selection effects remains unknown.","topicIds":["work-and-business--jobs-and-occupations","job-displacement-and-creation"],"domain":"other","claimType":"attributed reading","limits":"METR says selection effects, a lower pay rate, task differences, and time-reporting difficulties complicate interpretation; the experiment is not presented as a reliable measure of productivity impact. Text as retrieved on October 23, 2026; originally published February 24, 2026.","speaker":{"name":"METR","roleAtSource":"Publisher"},"source":{"title":"We are Changing our Developer Productivity Experiment Design","publisher":"METR","format":"Article","url":"https://metr.org/blog/2026-02-24-uplift-update/","publishedAt":"2026-02-24T08:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"METR says selection effects make its new experiment an unreliable signal of AI’s current productivity effect, particularly because developers who prefer working with AI may not participate.","excerpt":"we believe that the data from our new experiment gives us an unreliable signal of the current productivity effect of AI tools. The primary reason is that we have observed a significant increase in developers choosing not to participate","url":"https://metr.org/blog/2026-02-24-uplift-update/","relationship":"Source-stated"},{"text":"METR says its experiment may miss tasks with high expected uplift from AI because some developers chose not to submit tasks they did not want to do without AI.","excerpt":"30% to 50% of developers told us that they were choosing not to submit some tasks because they did not want to do them without AI. This implies we are systematically missing tasks which have high expected uplift from AI.","url":"https://metr.org/blog/2026-02-24-uplift-update/","relationship":"Source-stated"},{"text":"METR says selection effects make its estimate weak evidence about the size of any increase, and calls the central estimate likely a bad proxy for real productivity impact on these developers.","excerpt":"our data is only very weak evidence for the size of this increase.","url":"https://metr.org/blog/2026-02-24-uplift-update/","relationship":"Source-stated"},{"text":"METR plans to explore several other ways to study developer productivity, while noting limitations in some of those approaches.","excerpt":"we plan to continue exploring the following lines of research:","url":"https://metr.org/blog/2026-02-24-uplift-update/","relationship":"Source-stated"},{"text":"METR describes developer productivity as an important input to AI R&D acceleration.","excerpt":"The impact of AI tools on developer productivity is an important input to AI R&D acceleration","url":"https://metr.org/blog/2026-02-24-uplift-update/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2026-02-24-uplift-update/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How much would higher pay or shorter experiments reduce the selection effects METR identifies?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"METR says its developer productivity estimate is a bad proxy","summary":"METR’s page was published on February 24, 2026. METR says selection effects make its new experiment a poor measure of AI’s productivity impact on participating open-source developers. Developers who expect AI to help them more are less likely to join or to submit tasks they would rather do with AI; METR says this likely makes its estimate a lower bound. A lower pay rate may also have contributed to selection, and time measurements can be unreliable when developers use multiple AI agents concurrently. METR says the data offer only very weak evidence about the size of any increase in productivity, despite raw results showing some evidence for speedup. It calls the experiment’s central estimate likely a bad proxy for the real productivity impact of AI tools on these developers. METR says developer productivity is an important input to AI R&D acceleration and plans to explore other approaches, including shorter, more intensive experiments, observational data, questionnaires, fixed-task experiments, evaluations and developer-level experiments. Those approaches have limitations, and METR expects to keep redesigning parts of its measurement techniques as model capabilities increase. How much higher pay or shorter experiments would reduce the selection effects remains unknown.","question":"How much would higher pay or shorter experiments reduce the selection effects METR identifies?","claims":[{"text":"METR says selection effects make its new experiment an unreliable signal of AI’s current productivity effect, particularly because developers who prefer working with AI may not participate.","excerpt":"we believe that the data from our new experiment gives us an unreliable signal of the current productivity effect of AI tools. The primary reason is that we have observed a significant increase in developers choosing not to participate","support":"source-stated"},{"text":"METR says its experiment may miss tasks with high expected uplift from AI because some developers chose not to submit tasks they did not want to do without AI.","excerpt":"30% to 50% of developers told us that they were choosing not to submit some tasks because they did not want to do them without AI. This implies we are systematically missing tasks which have high expected uplift from AI.","support":"source-stated"},{"text":"METR says selection effects make its estimate weak evidence about the size of any increase, and calls the central estimate likely a bad proxy for real productivity impact on these developers.","excerpt":"our data is only very weak evidence for the size of this increase.","support":"source-stated"},{"text":"METR plans to explore several other ways to study developer productivity, while noting limitations in some of those approaches.","excerpt":"we plan to continue exploring the following lines of research:","support":"source-stated"},{"text":"METR describes developer productivity as an important input to AI R&D acceleration.","excerpt":"The impact of AI tools on developer productivity is an important input to AI R&D acceleration","support":"source-stated"}],"limitations":["METR says selection effects, a lower pay rate, task differences, and time-reporting difficulties complicate interpretation; the experiment is not presented as a reliable measure of productivity impact.","Text as retrieved on October 23, 2026; originally published February 24, 2026."],"mediaId":null,"correctionReason":null,"id":"54d658ff870e4bbb0638a847bac138a3b543d42ac48e761c915c570500f1cc5c","publicationId":"0d1ef902-2393-9301-fb20-07f24a4df975","version":1,"sourceUrl":"https://metr.org/blog/2026-02-24-uplift-update/","sourceTitle":"We are Changing our Developer Productivity Experiment Design","publisher":"METR","publishedAt":"2026-02-24T08:00:00.000Z","originalPublished":{"value":"2026-02-24T08:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T03:05:27.965Z","createdAt":"2026-11-04T20:41:43.395Z","historical":true,"entityIds":["metr"],"topicIds":["work-and-business--jobs-and-occupations","job-displacement-and-creation"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2026-02-24-uplift-update/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"c7d917da9660daf96f1eef9263a2f975bb25238d17fac4fd732fac8601bcccd9","reasonCodes":["bridge_screen"],"rubricVersion":"bridge-screen-v1","confidence":0.77,"provider":"typesafe","model":"jev-1.13.0","decidedAt":"2026-10-27T17:29:29.304Z"},"provenance":{"textHash":"81b6c9b7d710e11ed96cf12de5f307e8551454cfdcc9f6c66c84f0a657b9adfb","observationId":"b7cdc79213fc9c97869e0dcc80e290b4038dd4df8beaf6a5415e863249ec5fe3","retrievedAt":"2026-10-23T14:47:42.473Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"18ac224d-ea34-5d71-5d3d-c2eebae14e22"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"06ce2006-b1da-1094-d8e3-99484d8f3ec7"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-04T16:06:49.782Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/metr-says-its-developer-productivity-estimate-is-a-bad-proxy-54d658ff","formats":{"html":"/readings/metr-says-its-developer-productivity-estimate-is-a-bad-proxy-54d658ff","markdown":"/readings/metr-says-its-developer-productivity-estimate-is-a-bad-proxy-54d658ff.md","json":"/readings/metr-says-its-developer-productivity-estimate-is-a-bad-proxy-54d658ff.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-ai-agent","name":"AI agent","explanation":"An AI system that chooses and carries out actions toward a goal, often using software tools.","referenceUrl":"https://www.ibm.com/think/topics/ai-agents"}],"entities":[{"id":"metr","name":"METR","kind":"organization","context":"METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.","referenceUrls":["https://metr.org/"],"path":"/entities/metr","json":"/entities/metr.json","markdown":"/entities/metr.md"}]},"researchQuestions":[{"id":"ai-and-work","title":"When does AI become a dependable colleague?","question":"When does an AI become a dependable colleague?","url":"/questions/ai-and-work","edition":"product-management-read-2026-11-04"},{"id":"ai-work-quality-and-rework","title":"Does AI reduce the effort needed to finish work well?","question":"Does AI reduce the total effort needed to produce work that meets the required standard?","url":"/questions/ai-work-quality-and-rework","edition":"product-management-read-2026-11-04"},{"id":"ai-delegation-reliability","title":"When can work be delegated without checking every step?","question":"When can work be delegated without checking every step?","url":"/questions/ai-delegation-reliability","edition":"product-management-read-2026-11-04"},{"id":"ai-deployment-returns","title":"When is an AI deployment worth its total cost?","question":"When does a useful AI deployment become worth its total cost?","url":"/questions/ai-deployment-returns","edition":"product-management-read-2026-11-04"}],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"54d658ff870e4bbb0638a847bac138a3b543d42ac48e761c915c570500f1cc5c","revision":"0d1ef902-2393-9301-fb20-07f24a4df975","revisionKind":"publication-id","title":"METR says its developer productivity estimate is a bad proxy","url":"https://escapemeridian.com/readings/metr-says-its-developer-productivity-estimate-is-a-bad-proxy-54d658ff"},"source":{"type":"source","id":"https://metr.org/blog/2026-02-24-uplift-update/","revision":"81b6c9b7d710e11ed96cf12de5f307e8551454cfdcc9f6c66c84f0a657b9adfb","revisionKind":"sha256-source-text","title":"We are Changing our Developer Productivity Experiment Design","url":"https://metr.org/blog/2026-02-24-uplift-update/"},"status":"available","items":[{"id":"reading-citation:amodei-fast-diffusion:3b9176ef7c6a259fbe18b442f8cbb8d3e1e78cab9accec9cd2caf5a1342776b3:evidence.2.url","from":{"type":"reading","id":"amodei-fast-diffusion","revision":"3b9176ef7c6a259fbe18b442f8cbb8d3e1e78cab9accec9cd2caf5a1342776b3","revisionKind":"sha256-record","title":"Dario Amodei: adoption can move fast and still take work","url":"https://escapemeridian.com/readings/amodei-fast-diffusion"},"to":{"type":"source","id":"https://metr.org/blog/2026-02-24-uplift-update/","revision":null,"revisionKind":null,"title":"We are Changing our Developer Productivity Experiment Design","url":"https://metr.org/blog/2026-02-24-uplift-update/"},"relationship":"references","provenance":{"kind":"recorded-citation","recordId":"3b9176ef7c6a259fbe18b442f8cbb8d3e1e78cab9accec9cd2caf5a1342776b3","field":"evidence.2.url","recordedAt":null,"author":{"id":null,"name":"Escape Meridian","kind":"editorial"},"citedUrl":"https://metr.org/blog/2026-02-24-uplift-update/","assertedRelationship":"qualifies"}}],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"54d658ff870e4bbb0638a847bac138a3b543d42ac48e761c915c570500f1cc5c","title":"METR says its developer productivity estimate is a bad proxy","argument":"METR’s page was published on February 24, 2026. METR says selection effects make its new experiment a poor measure of AI’s productivity impact on participating open-source developers. Developers who expect AI to help them more are less likely to join or to submit tasks they would rather do with AI; METR says this likely makes its estimate a lower bound. A lower pay rate may also have contributed to selection, and time measurements can be unreliable when developers use multiple AI agents concurrently. METR says the data offer only very weak evidence about the size of any increase in productivity, despite raw results showing some evidence for speedup. It calls the experiment’s central estimate likely a bad proxy for the real productivity impact of AI tools on these developers. METR says developer productivity is an important input to AI R&D acceleration and plans to explore other approaches, including shorter, more intensive experiments, observational data, questionnaires, fixed-task experiments, evaluations and developer-level experiments. Those approaches have limitations, and METR expects to keep redesigning parts of its measurement techniques as model capabilities increase. How much higher pay or shorter experiments would reduce the selection effects remains unknown.","topicIds":["work-and-business--jobs-and-occupations","job-displacement-and-creation"],"domain":"other","claimType":"attributed reading","limits":"METR says selection effects, a lower pay rate, task differences, and time-reporting difficulties complicate interpretation; the experiment is not presented as a reliable measure of productivity impact. Text as retrieved on October 23, 2026; originally published February 24, 2026.","speaker":{"name":"METR","roleAtSource":"Publisher"},"source":{"title":"We are Changing our Developer Productivity Experiment Design","publisher":"METR","format":"Article","url":"https://metr.org/blog/2026-02-24-uplift-update/","publishedAt":"2026-02-24T08:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"METR says selection effects make its new experiment an unreliable signal of AI’s current productivity effect, particularly because developers who prefer working with AI may not participate.","excerpt":"we believe that the data from our new experiment gives us an unreliable signal of the current productivity effect of AI tools. The primary reason is that we have observed a significant increase in developers choosing not to participate","url":"https://metr.org/blog/2026-02-24-uplift-update/","relationship":"Source-stated"},{"text":"METR says its experiment may miss tasks with high expected uplift from AI because some developers chose not to submit tasks they did not want to do without AI.","excerpt":"30% to 50% of developers told us that they were choosing not to submit some tasks because they did not want to do them without AI. This implies we are systematically missing tasks which have high expected uplift from AI.","url":"https://metr.org/blog/2026-02-24-uplift-update/","relationship":"Source-stated"},{"text":"METR says selection effects make its estimate weak evidence about the size of any increase, and calls the central estimate likely a bad proxy for real productivity impact on these developers.","excerpt":"our data is only very weak evidence for the size of this increase.","url":"https://metr.org/blog/2026-02-24-uplift-update/","relationship":"Source-stated"},{"text":"METR plans to explore several other ways to study developer productivity, while noting limitations in some of those approaches.","excerpt":"we plan to continue exploring the following lines of research:","url":"https://metr.org/blog/2026-02-24-uplift-update/","relationship":"Source-stated"},{"text":"METR describes developer productivity as an important input to AI R&D acceleration.","excerpt":"The impact of AI tools on developer productivity is an important input to AI R&D acceleration","url":"https://metr.org/blog/2026-02-24-uplift-update/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2026-02-24-uplift-update/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How much would higher pay or shorter experiments reduce the selection effects METR identifies?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"METR says its developer productivity estimate is a bad proxy","summary":"METR’s page was published on February 24, 2026. METR says selection effects make its new experiment a poor measure of AI’s productivity impact on participating open-source developers. Developers who expect AI to help them more are less likely to join or to submit tasks they would rather do with AI; METR says this likely makes its estimate a lower bound. A lower pay rate may also have contributed to selection, and time measurements can be unreliable when developers use multiple AI agents concurrently. METR says the data offer only very weak evidence about the size of any increase in productivity, despite raw results showing some evidence for speedup. It calls the experiment’s central estimate likely a bad proxy for the real productivity impact of AI tools on these developers. METR says developer productivity is an important input to AI R&D acceleration and plans to explore other approaches, including shorter, more intensive experiments, observational data, questionnaires, fixed-task experiments, evaluations and developer-level experiments. Those approaches have limitations, and METR expects to keep redesigning parts of its measurement techniques as model capabilities increase. How much higher pay or shorter experiments would reduce the selection effects remains unknown.","question":"How much would higher pay or shorter experiments reduce the selection effects METR identifies?","claims":[{"text":"METR says selection effects make its new experiment an unreliable signal of AI’s current productivity effect, particularly because developers who prefer working with AI may not participate.","excerpt":"we believe that the data from our new experiment gives us an unreliable signal of the current productivity effect of AI tools. The primary reason is that we have observed a significant increase in developers choosing not to participate","support":"source-stated"},{"text":"METR says its experiment may miss tasks with high expected uplift from AI because some developers chose not to submit tasks they did not want to do without AI.","excerpt":"30% to 50% of developers told us that they were choosing not to submit some tasks because they did not want to do them without AI. This implies we are systematically missing tasks which have high expected uplift from AI.","support":"source-stated"},{"text":"METR says selection effects make its estimate weak evidence about the size of any increase, and calls the central estimate likely a bad proxy for real productivity impact on these developers.","excerpt":"our data is only very weak evidence for the size of this increase.","support":"source-stated"},{"text":"METR plans to explore several other ways to study developer productivity, while noting limitations in some of those approaches.","excerpt":"we plan to continue exploring the following lines of research:","support":"source-stated"},{"text":"METR describes developer productivity as an important input to AI R&D acceleration.","excerpt":"The impact of AI tools on developer productivity is an important input to AI R&D acceleration","support":"source-stated"}],"limitations":["METR says selection effects, a lower pay rate, task differences, and time-reporting difficulties complicate interpretation; the experiment is not presented as a reliable measure of productivity impact.","Text as retrieved on October 23, 2026; originally published February 24, 2026."],"mediaId":null,"correctionReason":null,"id":"54d658ff870e4bbb0638a847bac138a3b543d42ac48e761c915c570500f1cc5c","publicationId":"0d1ef902-2393-9301-fb20-07f24a4df975","version":1,"sourceUrl":"https://metr.org/blog/2026-02-24-uplift-update/","sourceTitle":"We are Changing our Developer Productivity Experiment Design","publisher":"METR","publishedAt":"2026-02-24T08:00:00.000Z","originalPublished":{"value":"2026-02-24T08:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T03:05:27.965Z","createdAt":"2026-11-04T20:41:43.395Z","historical":true,"entityIds":["metr"],"topicIds":["work-and-business--jobs-and-occupations","job-displacement-and-creation"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2026-02-24-uplift-update/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"c7d917da9660daf96f1eef9263a2f975bb25238d17fac4fd732fac8601bcccd9","reasonCodes":["bridge_screen"],"rubricVersion":"bridge-screen-v1","confidence":0.77,"provider":"typesafe","model":"jev-1.13.0","decidedAt":"2026-10-27T17:29:29.304Z"},"provenance":{"textHash":"81b6c9b7d710e11ed96cf12de5f307e8551454cfdcc9f6c66c84f0a657b9adfb","observationId":"b7cdc79213fc9c97869e0dcc80e290b4038dd4df8beaf6a5415e863249ec5fe3","retrievedAt":"2026-10-23T14:47:42.473Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"18ac224d-ea34-5d71-5d3d-c2eebae14e22"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"06ce2006-b1da-1094-d8e3-99484d8f3ec7"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-04T16:06:49.782Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"0d1ef902-2393-9301-fb20-07f24a4df975","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"This is a single METR blog post with its own argument. The reading attributes its claims to METR throughout, and none are presented as established fact or Escape Meridian's view.","excerpt":"we believe that the data from our new experiment gives us an unreliable signal of the current productivity effect of AI tools"},{"dimension":"summary","status":"uncertain","explanation":"The reading omits what changed. It never says METR's earlier study found AI caused a roughly 20% slowdown, or that METR now believes developers are likely more sped up in early 2026. Writing 'weak evidence about the size of any increase' turns METR's stated belief in an increase into a vague possibility. Smallest repair: add one sentence on the prior slowdown finding and METR's belief that speedup is now likely larger. Separately, 'Those approaches have limitations' overgeneralizes; the source flags limitations for some approaches, such as questionnaires and developer-level experiments, not all.","excerpt":"we believe it is likely that developers are more sped up from AI tools now — in early 2026 — compared to our estimates from early 2025"},{"dimension":"date","status":"pass","explanation":"The reading states February 24, 2026, which matches the UTC day of originalPublished. It uses no later-dated material or hindsight framing. Kind is publisher-page, so no archive disclosure is needed.","excerpt":null},{"dimension":"support","status":"pass","explanation":"The claims on non-participation, task selection, the lower-bound estimate, the lower pay rate, unreliable time measurement with concurrent agents, the bad-proxy judgment, and the planned alternative methods are all directly stated in the source. The question follows from METR's note that shorter, more intensive experiments and possibly higher pay could alleviate selection.","excerpt":"The selection issues would be alleviated if we had higher compliance, which could be achieved if we run shorter intense experiments, and possibly pay higher rates."},{"dimension":"media","status":"pass","explanation":"No media was selected, and no verified candidates were supplied. This is appropriate.","excerpt":null}],"createdAt":"2026-11-04T16:06:49.782Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}