{"targetId":"ai-work-quality-and-rework","coveredDimensionIds":["software-development","customer-support","writing-analysis"],"status":"current","revisionId":"57cdcc2b517a602ecb395e140296d06308bd7c8a11e9e2a674ed538f781a14c6","version":7,"judgment":"unchanged","reason":"The guide already advises using AI to investigate failures while the developer keeps responsibility for the diagnosis, and it counts checking and repair in total effort. This arrival adds the first deployment-scale evidence on AI assisting the repair side itself: 90.14% root-cause accuracy on 71 failures and few “Not helpful” ratings across 52,635 failing tests. That supports the investigation task. However, the arrival measures accuracy and perception, not effort at fixed quality or defects after handoff. It therefore does not satisfy any signpost or revision trigger, does not shift the 2027–2028 horizon, and does not weaken the software-repair risk path. A roughly 10% error rate in the sample also reinforces the existing advice to check suggestions. Adopting it as an exact dependency for the software-development investigation task is warranted without changing any judgment.","updatedAt":"2026-11-20T17:24:45.824Z","history":"/guide-maintenance/ai-work-quality-and-rework/history.json","authorship":{"provider":"anthropic","model":"claude-opus-5-5","promptVersion":"guide-maintain-v4","receiptKey":"bfr:edit:d6a2230ba2e7cde4ae05eba83b73b636f451832d15d155c5da928062ff1f9a4a"},"evidence":[{"id":"c35238e26e0ee9c0b528e0ae014b9e92b371cc19b516baaf346e61ab9e97bb95","source":{"id":"67fd980a191d98ac05fce306aff334d53cc1697d1aafd06528c755a2734cbd5e","workId":"18f3c3ca31491462a8ab52f797ebadd2a7087803317e69a5152e2a0c4cfbed65","publicationId":"f42f9a06-37c6-4173-a64b-cb1b7c9e1aef","lane":"bridge","textHash":"8ea3533aa08b1f63c143b0a383911da93e88f464edd2d8e61edebf2a97a42232","textBasis":"source-text","clocks":{"eventTime":{"value":null,"precision":"unknown"},"sourcePublished":{"value":"2026-10-16T00:00:00.000Z","precision":"instant","basis":"backfill:article_published"},"availableBasis":"retained-version-observation","availableAt":"2026-10-28T02:31:50.267Z","observedAt":"2026-10-18T02:23:16.572Z","authoredAt":"2026-10-29T14:59:05.929Z","publishedAt":"2026-10-29T15:58:18.797Z","indexedAt":"2026-10-31T20:24:00.902Z"},"eligibleWhenAdopted":true,"currentlyEligible":true,"eligibilityReason":"canonical_public_reading","title":"How employment is changing at Anthropic","url":"https://www.anthropic.com/research/employment-at-anthropic"},"association":{"decision":"linked","relationship":"direct","role":"challenges","explanation":"This arrival is the full retained text of the foundation source anthropic-workforce-2026: the same workId and textHash. Its anth-bottleneck finding is already considered. It adds one relevant finding the guide does not yet rely on. The authors compared agent-written and human-written code by the same engineer, in the same month and directory. On lines later removed or reverted, they found no difference. This bears on the software-repair risk path and its signpost, and on the review-absorbs-gains signpost that time saved returns as correction. At one firm, on these proxies, agent code did not generate visibly more later repair. The challenge is limited. The authors warn that AI errors may be harder to identify. Adoption is not random, and adopters were already on a rising trajectory. The comparison measures code churn proxies, not total human effort at a fixed required quality. The same report finds that agent management and other meta-work rose to 22% of interactive sessions. It also finds that reading all agent output would take over half the work week for non-technical groups. That cuts the other way: the verification and attention cost the guide asks readers to count is substantial. Our read is that the evidence narrows where repair costs show up rather than showing that full-work effort fell.","targetIds":["software-repair","software-repair:signpost","quality-review-absorbs-gains:signpost:1","quality-review-absorbs-gains:condition:0"],"spans":[{"sourceId":"67fd980a191d98ac05fce306aff334d53cc1697d1aafd06528c755a2734cbd5e","excerpt":"We compare code committed by the same employee, in the same month, in the same coding directory, but written by coding agents rather than written directly by the human employee."},{"sourceId":"67fd980a191d98ac05fce306aff334d53cc1697d1aafd06528c755a2734cbd5e","excerpt":"There are many dimensions of code quality, and AI errors may be harder to identify than human ones. But we find no difference in rates of lines of code subsequently removed or reverted."},{"sourceId":"67fd980a191d98ac05fce306aff334d53cc1697d1aafd06528c755a2734cbd5e","excerpt":"Those who adopt were already on a steeply upward trajectory compared to non-adopters."},{"sourceId":"67fd980a191d98ac05fce306aff334d53cc1697d1aafd06528c755a2734cbd5e","excerpt":"Much of this rise was due specifically to agent management: writing specs for multi-agent runs, looping one agent to review another’s output, and monitoring pipelines."}],"mechanism":null,"assumptions":["Deletion within 90 days, bug-fix deletion, reverts, lint suppressions, fix-me notes and skipped tests are partial proxies for downstream repair, not a full measure of defects after handoff.","Within-person, same-month, same-directory comparison controls some but not all differences between agent-written and human-written tasks; task assignment to agents may differ.","Anthropic is one AI company with unusual tooling and incentives; results may not transfer to other employers or developers."],"intermediateSourceIds":[],"familyId":"text:8ea3533aa08b1f63c143b0a383911da93e88f464edd2d8e61edebf2a97a42232","duplicateOf":null},"references":[{"id":"67fd980a191d98ac05fce306aff334d53cc1697d1aafd06528c755a2734cbd5e","title":"How employment is changing at Anthropic","url":"https://www.anthropic.com/research/employment-at-anthropic"}]},{"id":"65cbbc280ec6a4af0d0641aa02732811d5709436e6cdc8961caf8d70cb9d4ecc","source":{"id":"a825216b2058278a035277d9cec10ef4854bfb3e7e34f94877394b90a39d32ca","workId":"31600a8ceb52a83f8aa5dd75426ffd6ba753517638069817b442d5986c391a06","publicationId":"5a717d18-d44f-c924-6cf8-64016cd1901c","lane":"bridge","textHash":"a35f4653858a02fa48a9001148e323c4df710e869f8163a29e535e0cbc8f88ff","textBasis":"source-text","clocks":{"eventTime":{"value":null,"precision":"unknown"},"sourcePublished":{"value":"2026-06-11T02:29:51.000Z","precision":"instant","basis":"backfill:feed_published"},"availableBasis":"retained-version-observation","availableAt":"2026-10-23T20:33:36.887Z","observedAt":"2026-10-23T20:31:49.340Z","authoredAt":"2026-10-29T18:09:13.093Z","publishedAt":"2026-10-30T07:48:56.471Z","indexedAt":"2026-10-31T21:10:10.364Z"},"eligibleWhenAdopted":true,"currentlyEligible":true,"eligibilityReason":"canonical_public_reading","title":"Why AI hasn’t replaced software engineers, and won’t","url":"https://www.normaltech.ai/p/why-ai-hasnt-replaced-software-engineers"},"association":{"decision":"linked","relationship":"direct","role":"context","explanation":"The retrieval match on coding and debugging leads to substantive software evidence. Narayanan and Kapoor’s June 2026 essay is mainly an argument about layoffs and labor demand. That part does not bear on this guide’s outcome, which is total effort at fixed quality. It does, however, report two studies on how agent-generated code becomes delivered work, and it offers an attributed account of that process. Both figures come second-hand through the essay. We have not read either underlying study.\n\nFirst, the essay reports that in a study of 100,000 GitHub developers, an eightfold increase in lines written produced only 30% more releases. Second, it reports that SWE-chat found only 44% of agent-produced code survived into user commits, with higher vulnerability rates for vibe-coded commits. The authors call that dataset self-selected.\n\nThis bears on the software-repair risk path and its signpost, which asks whether generation accelerates while integration, failed changes or later repair grow. It shows generation far outrunning releases. It does not show that integration time or later repair grew, so the signpost is not met.\n\nIt also bears on software-whole-tasks and its condition that developers retain enough understanding to maintain the result. The authors argue that deciding, verifying and understanding remain human bottlenecks, and that supervising agents is “surprisingly time consuming.” It bears as well on quality-review-absorbs-gains:signpost:1, the signpost that time saved at generation returns as integration, correction or customer support.\n\nThe release figure cuts both ways. A 30% rise in releases is still a gain in shipped output, and nothing shows that quality fell or effort rose. That fits the guide’s existing position: count delivery and repair, and treat generated output as no measure of finished work. It does not establish either the risk path or the rival account.\n\nThis evidence is distinct from the Anthropic churn finding and is not a duplicate of it.","targetIds":["software-repair","software-repair:signpost","software-whole-tasks","software-whole-tasks:condition:1","quality-review-absorbs-gains:signpost:1"],"spans":[{"sourceId":"a825216b2058278a035277d9cec10ef4854bfb3e7e34f94877394b90a39d32ca","excerpt":"AI agents led to an eight-fold increase in the number of lines of code written, consistent with the idea that AI almost completely compresses the Execute layer of the sandwich. But this led to only 30% more releases"},{"sourceId":"a825216b2058278a035277d9cec10ef4854bfb3e7e34f94877394b90a39d32ca","excerpt":"only 44% of agent-produced code survives into user commits, that vibe-coded commits introduce vulnerabilities at nine times the human-only rate"},{"sourceId":"a825216b2058278a035277d9cec10ef4854bfb3e7e34f94877394b90a39d32ca","excerpt":"The self-selected nature of the dataset means that we can’t draw strong conclusions based on this study alone"},{"sourceId":"a825216b2058278a035277d9cec10ef4854bfb3e7e34f94877394b90a39d32ca","excerpt":"engineers are discovering that supervising coding agents is surprisingly time consuming"}],"mechanism":null,"assumptions":["The essay accurately reports the “Writing Code vs. Shipping Code” and SWE-chat results; the underlying studies have not been read here.","Releases are a proxy for delivered work, not for total human effort at a fixed required quality.","The SWE-chat sample is self-selected open-source developers and may not transfer to other teams."],"intermediateSourceIds":[],"familyId":"text:a35f4653858a02fa48a9001148e323c4df710e869f8163a29e535e0cbc8f88ff","duplicateOf":null},"references":[{"id":"a825216b2058278a035277d9cec10ef4854bfb3e7e34f94877394b90a39d32ca","title":"Why AI hasn’t replaced software engineers, and won’t","url":"https://www.normaltech.ai/p/why-ai-hasnt-replaced-software-engineers"}]},{"id":"b8e726c916bda5d276db573873f2cf131c49d1785a5722e7d66727c47541d915","source":{"id":"354e93751d2002b74d7602a2e45dec7ab28a707029523e21afca5fa744a6603f","workId":"0d9f43c6ab3dd33af6764c11f2337d207783564e1f850b909c941169d0106ea1","publicationId":"675eaa23-c5ad-4504-8348-11f0b6972a8e","lane":"live","textHash":"1bb5aefbaf44086e3ecaded2d45219cb9dd29700dd005839a2bf9abf6129cdbe","textBasis":"source-text","clocks":{"eventTime":{"value":null,"precision":"unknown"},"sourcePublished":{"value":"2023-02-13","precision":"day","basis":"publisher_citation_date"},"availableBasis":"retained-version-observation","availableAt":"2026-11-20T12:26:50.782Z","observedAt":"2026-11-20T03:27:16.171Z","authoredAt":"2026-11-20T12:27:07.894Z","publishedAt":"2026-11-20T12:27:28.052Z","indexedAt":"2026-11-20T12:38:14.735Z"},"eligibleWhenAdopted":true,"currentlyEligible":true,"eligibilityReason":"canonical_public_reading","title":"The Impact of AI on Developer Productivity: Evidence from GitHub Copilot","url":"https://arxiv.org/abs/2302.06590"},"association":{"decision":"linked","relationship":"direct","role":"supports","explanation":"The arrival is the arXiv abstract of Peng, Kalliamvakou, Cihon and Demirer's controlled GitHub Copilot experiment, submitted 13 February 2023. It is a new evidence family, not a report of any foundation or adopted study. It has the same population, task and outcome type as the guide: developers, coding, and comparative time to complete work.\n\nWhat the source observes: recruited developers implemented one specified HTTP server in JavaScript, and the group with AI access completed it 55.8% faster than the control group. This is a comparative measurement, not a self-report.\n\nWhat it bears on: the main question's claim that AI saves effort where the task is defined and checkable. It also bears on quality-useful-work-expands and on software-whole-tasks, which concerns delegating complete, well-specified changes. Together with METR's early-2025 slowdown in familiar, mature repositories, it fits the guide's scope distinction. Gains appear on a bounded new task and can reverse on complex familiar work.\n\nLimits: the abstract measures speed on a single task with early-2023 tools. It does not state the quality standard or acceptance check, and it does not count later review, integration or repair. So it does not meet the quality-useful-work-expands signpost of lower total effort at stable quality. Nor does it meet the revision trigger asking for defects after handoff.\n\nThe authors' remark about career transitions is their interpretation of heterogeneous effects. The abstract reports no career outcomes, so we do not link it to entry or progression.\n\nThe strongest rival is the guide's first alternative: favorable task selection explains apparent gains. A single greenfield task chosen for the experiment is exactly the kind of setting that rival predicts will flatter AI.","targetIds":["ai-work-quality-and-rework","quality-useful-work-expands","software-whole-tasks"],"spans":[{"sourceId":"354e93751d2002b74d7602a2e45dec7ab28a707029523e21afca5fa744a6603f","excerpt":"Recruited software developers were asked to implement an HTTP server in JavaScript as quickly as possible."},{"sourceId":"354e93751d2002b74d7602a2e45dec7ab28a707029523e21afca5fa744a6603f","excerpt":"The treatment group, with access to the AI pair programmer, completed the task 55.8% faster than the control group."}],"mechanism":null,"assumptions":["Completion time on one standardized task is a partial measure of effort; it excludes later review, integration and repair.","The abstract does not state how completion or correctness was judged, so fixed quality cannot be confirmed from the available text.","Early-2023 tools and recruited developers on a new, bounded task may not transfer to familiar codebases or later systems."],"intermediateSourceIds":[],"familyId":"text:1bb5aefbaf44086e3ecaded2d45219cb9dd29700dd005839a2bf9abf6129cdbe","duplicateOf":null},"references":[{"id":"354e93751d2002b74d7602a2e45dec7ab28a707029523e21afca5fa744a6603f","title":"The Impact of AI on Developer Productivity: Evidence from GitHub Copilot","url":"https://arxiv.org/abs/2302.06590"}]},{"id":"2c3f846a68fd97e669aa50f7edc9e6d70432b200b134931171a3be567fa5cce3","source":{"id":"8292123889c408d14c0ef026f248422432bf6650cd749e0112dc6f5037405b37","workId":"9850bcabd3c2776c23653d367680367b2403dbafc9346d9de57182f9d6c20199","publicationId":"56616587-5dc2-48b0-86bc-578f02583c7d","lane":"live","textHash":"0f2c37992c0ce8768bb5b809aab0f5304c0c699ec67a36badbc749a80f1385a5","textBasis":"source-text","clocks":{"eventTime":{"value":null,"precision":"unknown"},"sourcePublished":{"value":"2026-07-27","precision":"day","basis":"publisher_citation_date"},"availableBasis":"retained-version-observation","availableAt":"2026-11-20T13:52:04.014Z","observedAt":"2026-11-20T13:50:23.739Z","authoredAt":"2026-11-20T13:52:44.881Z","publishedAt":"2026-11-20T13:53:35.151Z","indexedAt":"2026-11-20T13:53:38.906Z"},"eligibleWhenAdopted":true,"currentlyEligible":true,"eligibilityReason":"canonical_public_reading","title":"Verification-Conditioned Use: A Qualitative Study on How Generative AI Reshapes Learning, Autonomy, and Market Entry for Junior Software Developers","url":"https://arxiv.org/abs/2607.24606"},"association":{"decision":"linked","relationship":"direct","role":"limits","explanation":"Retrieval came through the writing-verification route (writing-polished-error, writing-more-judgment). However, the arrival is about software newcomers, not writing or analysis, so it is not linked to those writing paths. What it does reach is the software-development dimension.\n\nSource: Andriotte and Ribeiro, arXiv v1, submitted 27 July 2026. Only the abstract is available. The study is qualitative. The authors interviewed 13 interns and junior developers and used Braun and Clarke thematic analysis. It is neither a comparative measurement nor a measure of effort, quality or career outcomes.\n\nWhat participants report: the criterion that most often decides between AI and manual work is \"the ability to check the result\". They describe feeling more capable but less in ownership of the result.\n\nWhat the authors argue: a \"formative paradox\". In their words, shallow learning induced by AI \"makes it harder\" to build critical-judgment competence. That is the authors' interpretation of interview themes, not a measured learning deficit.\n\nWhere it bears on the guide:\n(1) software-whole-tasks:condition:1, \"Developers retain enough understanding to maintain the result.\" For newcomers, the reported loss of ownership and the authors' shallow-learning argument suggest the condition cannot be assumed. Holding it may require the deliberate practices participants describe. This limits the favored path for juniors without refuting it.\n(2) software-repair:condition:1, \"maintainers cannot explain the new code.\" The arrival names a plausible route by which that condition arises among people entering the field.\n(3) The software-development dimension's who field already says \"Being able to generate a change is not yet being able to maintain it.\" The participants' self-described sustainable practices fit the dimension's advice to review, test and understand: \"reviewing before accepting, refusing to use AI without understanding it\".\n(4) ai-work-quality-and-rework:assumption:1, that tasks can be selected rather than forced through one AI process. Newcomers report already selecting tasks by checkability. That is consistent with the guide's scope, but it is self-reported usage, not evidence that such selection lowers total effort.\n\nThe strongest rival reading is that the paradox reflects the novice stage itself, or participants' self-perception, rather than a lasting competence gap. Thirteen interviews cannot distinguish the two. A longitudinal comparison of newcomers' independent debugging or maintenance ability with and without deliberate unassisted practice would.","targetIds":["software-whole-tasks:condition:1","software-repair:condition:1","software-development","ai-work-quality-and-rework:assumption:1"],"spans":[{"sourceId":"8292123889c408d14c0ef026f248422432bf6650cd749e0112dc6f5037405b37","excerpt":"thirteen interns and junior developers were interviewed individually, by videoconference."},{"sourceId":"8292123889c408d14c0ef026f248422432bf6650cd749e0112dc6f5037405b37","excerpt":"the criterion that most often decides between AI and manual work is not deadline or task complexity, but the ability to check the result."},{"sourceId":"8292123889c408d14c0ef026f248422432bf6650cd749e0112dc6f5037405b37","excerpt":"the autonomy paradox (feeling more capable yet less in ownership of the result)"},{"sourceId":"8292123889c408d14c0ef026f248422432bf6650cd749e0112dc6f5037405b37","excerpt":"the shallow learning that AI induces makes it harder to build the very critical-judgment competence that, according to participants, the market has begun to demand."},{"sourceId":"8292123889c408d14c0ef026f248422432bf6650cd749e0112dc6f5037405b37","excerpt":"reviewing before accepting, refusing to use AI without understanding it, asking the tool for explanations, and keeping deliberate practice outside of AI-assisted work."}],"mechanism":null,"assumptions":["Interview self-perceptions of reduced ownership indicate some real risk to newcomers' understanding; this has not been measured in the study.","Only the abstract is available. Sampling, setting and tools are not described there, and nothing is inferred from omitted text.","Thirteen interns and junior developers may not represent other newcomers, employers or later tools."],"intermediateSourceIds":[],"familyId":"text:0f2c37992c0ce8768bb5b809aab0f5304c0c699ec67a36badbc749a80f1385a5","duplicateOf":null},"references":[{"id":"8292123889c408d14c0ef026f248422432bf6650cd749e0112dc6f5037405b37","title":"Verification-Conditioned Use: A Qualitative Study on How Generative AI Reshapes Learning, Autonomy, and Market Entry for Junior Software Developers","url":"https://arxiv.org/abs/2607.24606"}]},{"id":"b0bf2b939800ac0e7fb477c36b3ec7023f440f6816554431bf6fe0c097cfd75f","source":{"id":"c4feae2c9f689cadb52cf86f4c02473e75d64a95424b1f231151d50db645c4e0","workId":"09b29f95e46a8f0bbc508732859a0ecc9370d497ffac4e63252b009e932677fc","publicationId":"63c2eb04-6e97-4e52-8372-9668ed6245a9","lane":"live","textHash":"dc32247ea63e69de4ada0f38f7b7a300652e7370de83c64ff309eca970a49d0b","textBasis":"source-text","clocks":{"eventTime":{"value":null,"precision":"unknown"},"sourcePublished":{"value":"2024-10-16","precision":"day","basis":"publisher_citation_date"},"availableBasis":"retained-version-observation","availableAt":"2026-11-20T13:57:54.444Z","observedAt":"2026-11-20T13:57:46.872Z","authoredAt":"2026-11-20T13:58:11.068Z","publishedAt":"2026-11-20T13:58:42.185Z","indexedAt":"2026-11-20T14:08:12.794Z"},"eligibleWhenAdopted":true,"currentlyEligible":true,"eligibilityReason":"canonical_public_reading","title":"How much does AI impact development speed? An enterprise-based randomized controlled trial","url":"https://arxiv.org/abs/2410.12944"},"association":{"decision":"linked","relationship":"direct","role":"supports","explanation":"The arrival is the arXiv v3 abstract (v1 submitted 16 October 2024; v3 revised 11 November 2024) of Paradis et al.'s randomized controlled trial. Its population (software engineers), task (development work) and outcome type (comparative time on task) fall within the guide's software scope. It is a new evidence family. It is not a report of METR, Peng et al., Anthropic or the Narayanan and Kapoor essay.\n\nWhat the source observes: in a randomized lab trial, 96 full-time Google engineers were given three AI features on one \"complex, enterprise-grade task\". AI \"significantly shortened\" their time on task. The authors' best estimate is about 21%, with a large confidence interval. Developers who spend more daily hours on code were faster with AI.\n\nWhat the authors argue: the effect size need not apply more broadly. They also say results from internal Google tooling in summer 2024 may not translate across tools or over time.\n\nWhere it bears:\n- The main question's claim that AI can lower effort on defined, checkable work.\n- quality-useful-work-expands.\n- software-whole-tasks, which concerns delegating well-specified changes.\n- quality-review-absorbs-gains:condition:1, the rival that selection of enthusiastic users or favorable tasks explains apparent gains. Randomization addresses participant selection. The setting is also a complex enterprise task rather than a small greenfield task, so this weakens the rival somewhat relative to the Peng et al. Copilot result.\n\nLimits: the outcome is time on task in a lab. The abstract states no quality standard or acceptance test, and it does not count later review, integration or defects after handoff. It therefore does not meet the signpost of lower total effort at stable quality or the revision trigger on defects after handoff.\n\nIt does not contradict METR's early-2025 19% slowdown. The settings differ: a researcher-set lab task with company tooling versus experienced developers working real issues in their own familiar repositories. That difference fits the guide's existing scope distinction rather than resolving it.","targetIds":["ai-work-quality-and-rework","quality-useful-work-expands","software-whole-tasks","quality-review-absorbs-gains:condition:1"],"spans":[{"sourceId":"c4feae2c9f689cadb52cf86f4c02473e75d64a95424b1f231151d50db645c4e0","excerpt":"Our randomized controlled trial with 96 full-time Google software engineers contributes to this literature by sharing an estimate of the impact of three AI features on the time developers spent on a complex, enterprise-grade task."},{"sourceId":"c4feae2c9f689cadb52cf86f4c02473e75d64a95424b1f231151d50db645c4e0","excerpt":"Our best estimate of the size of this effect, controlling for factors known to influence developer time on task, stands at about 21\\%, although our confidence interval is large."},{"sourceId":"c4feae2c9f689cadb52cf86f4c02473e75d64a95424b1f231151d50db645c4e0","excerpt":"we cannot assume that the effect size obtained in our lab study will necessarily apply more broadly"}],"mechanism":null,"assumptions":["Time on task in a lab trial is a partial measure of effort; it excludes later review, integration and repair.","The abstract does not state how task completion or correctness was judged, so fixed quality cannot be confirmed from the available text.","Internal Google tooling from summer 2024, used on one researcher-set task, may not transfer to other employers, familiar codebases or later tools."],"intermediateSourceIds":[],"familyId":"text:dc32247ea63e69de4ada0f38f7b7a300652e7370de83c64ff309eca970a49d0b","duplicateOf":null},"references":[{"id":"c4feae2c9f689cadb52cf86f4c02473e75d64a95424b1f231151d50db645c4e0","title":"How much does AI impact development speed? An enterprise-based randomized controlled trial","url":"https://arxiv.org/abs/2410.12944"}]},{"id":"33f74aa2a0cfd8b3f0364af4cf5687405989e7de367a180fbb6991cf337a16fa","source":{"id":"eb04831d7dd22846fe7bd91d6cf6edfc652d2bd063688f237081aed194e38bc4","workId":"6e31ccff2d84bf18ead31f2172b23f1f655a33195f67ac4600908c13c6b6edf3","publicationId":"36465bcd-07dc-46af-872a-8a487059ba08","lane":"live","textHash":"19b1137097aefd9a8d96777879a55892bb0245cefe9ce9dca1216eed96ba080c","textBasis":"source-text","clocks":{"eventTime":{"value":null,"precision":"unknown"},"sourcePublished":{"value":"2025-07-28","precision":"day","basis":"publisher_citation_date"},"availableBasis":"retained-version-observation","availableAt":"2026-11-20T16:42:05.877Z","observedAt":"2026-11-20T16:40:39.726Z","authoredAt":"2026-11-20T16:42:44.693Z","publishedAt":"2026-11-20T16:43:21.146Z","indexedAt":"2026-11-20T16:53:25.142Z"},"eligibleWhenAdopted":true,"currentlyEligible":true,"eligibilityReason":"canonical_public_reading","title":"\"Maybe We Need Some More Examples:\" Individual and Team Drivers of Developer GenAI Tool Use","url":"https://arxiv.org/abs/2507.21280"},"association":{"decision":"linked","relationship":"indirect","role":"limits","explanation":"Retrieval came through the whole-task-effort route. The source is substantive software evidence, though it does not measure the guide's outcome. It is the arXiv v1 abstract of Miller, Choudhuri, Ulloa, Haniyur, DeLine, Storey, Murphy-Hill, Bird and Butler, submitted 28 July 2025. It is a new evidence family, distinct from METR, Peng et al., Paradis et al., Anthropic, Narayanan and Kapoor, and Andriotte and Ribeiro.\n\nWhat the source observes: paired interviews with 54 developers across 27 teams, pairing one frequent and one infrequent user per team. The study is qualitative. It explains differences in usage, not effort, quality or defects.\n\nWhat the authors argue: usage differs mainly with how developers perceive the tool, how they engage with it and how they respond to challenges. They also argue that organizational expectations of rapid gains without investment in learning support create a \"Productivity Pressure Paradox\" that undermines the benefits motivating adoption. That paradox is the authors' implication from interview themes, not a measured productivity effect.\n\nWhy the link is indirect: the guide's base case depends on teams spreading gains by redesigning workflows during 2027–2028. It does so by reusing requirements and checking routines and by selecting tasks rather than forcing every task through one AI process (ai-work-quality-and-rework:assumption:1). The quality-useful-work-expands horizon likewise rests on \"task selection and redesign driving the spread.\" The arrival names a prerequisite for that redesign: developers must learn adaptive ways of working with the tool, and organizations must fund that learning rather than demand immediate gains. If that prerequisite fails, the spread could stall even where tasks are checkable. Read this way, the source qualifies the mechanism and timing of the favored path; it does not challenge whether savings occur at fixed quality.\n\nRival reading: frequent and infrequent users may differ because their tasks differ in how well AI suits them, not because of disposition or learning support. That would fit the guide's existing checkability scope rather than add a new condition. The abstract cannot separate the two accounts. A comparison of teams randomized to receive or not receive learning support, measuring total effort at fixed quality, would.","targetIds":["ai-work-quality-and-rework:assumption:1","quality-useful-work-expands"],"spans":[{"sourceId":"eb04831d7dd22846fe7bd91d6cf6edfc652d2bd063688f237081aed194e38bc4","excerpt":"Through paired interviews with 54 developers across 27 teams -- one frequent and one infrequent user per team"},{"sourceId":"eb04831d7dd22846fe7bd91d6cf6edfc652d2bd063688f237081aed194e38bc4","excerpt":"their engagement approach (experimental vs. conservative), and how they respond when encountering challenges (with adaptive persistence vs. quick abandonment)"},{"sourceId":"eb04831d7dd22846fe7bd91d6cf6edfc652d2bd063688f237081aed194e38bc4","excerpt":"widespread organizational expectations for rapid productivity gains without sufficient investment in learning support creates a \"Productivity Pressure Paradox,\" undermining the very productivity benefits that motivate adoption."}],"mechanism":"Organizational learning support and tolerance for experimentation shape whether developers adopt adaptive, persistent use. That adaptive use is a prerequisite for the task selection and workflow redesign through which the guide expects effort savings to spread during 2027–2028.","assumptions":["Adaptive, persistent use is necessary for the reuse of requirements and checking routines that the guide's base case relies on; the abstract does not measure this link.","The authors' paradox is an implication from interviews; no measured effort, quality or productivity outcome is reported in the available text.","Usage differences between paired frequent and infrequent users reflect perception and learning rather than mainly differences in task suitability.","Only the abstract is available; tools, employer setting and dates of interviews are not described there, and nothing is inferred from omitted text."],"intermediateSourceIds":[],"familyId":"text:19b1137097aefd9a8d96777879a55892bb0245cefe9ce9dca1216eed96ba080c","duplicateOf":null},"references":[{"id":"eb04831d7dd22846fe7bd91d6cf6edfc652d2bd063688f237081aed194e38bc4","title":"\"Maybe We Need Some More Examples:\" Individual and Team Drivers of Developer GenAI Tool Use","url":"https://arxiv.org/abs/2507.21280"}]},{"id":"b6af783bfe6d94b6e7584d7a1f9198f91f6763f54c5e98f6101e192436e32145","source":{"id":"a3328b3786e180a8fabbe945847cddaeea43d1a4d24a47fee6d4ec7bbad38f66","workId":"c5189029a128d8c1c18b47105b6e284f8c1a625d5deda9955d52832a7c093f76","publicationId":"40eb1c6b-df63-4c76-84ab-efd222d91180","lane":"live","textHash":"9d445b9b73bab7c912a9a9c29b6372bf3912c73a4af19a4a9a0c6424e586f806","textBasis":"source-text","clocks":{"eventTime":{"value":null,"precision":"unknown"},"sourcePublished":{"value":"2026-04-13","precision":"day","basis":"publisher_citation_date"},"availableBasis":"retained-version-observation","availableAt":"2026-11-20T16:43:22.012Z","observedAt":"2026-11-20T16:40:39.726Z","authoredAt":"2026-11-20T16:43:42.169Z","publishedAt":"2026-11-20T16:44:07.264Z","indexedAt":"2026-11-20T16:53:35.551Z"},"eligibleWhenAdopted":true,"currentlyEligible":true,"eligibilityReason":"canonical_public_reading","title":"LLM-Based Automated Diagnosis Of Integration Test Failures At Google","url":"https://arxiv.org/abs/2604.12108"},"association":{"decision":"linked","relationship":"direct","role":"supports","explanation":"Retrieval matched on “code review” and also on the writing-verification route. The writing match is incidental: the abstract concerns diagnosing software integration test failures, not writing or analysis, so it is not linked to writing-polished-error or writing-more-judgment.\n\nSource: Ziftci, Liu, Greene and Dalloro, arXiv v1, submitted 13 April 2026. This is a new evidence family, distinct from METR, Peng et al., Paradis et al., Anthropic, Narayanan and Kapoor, Andriotte and Ribeiro, and Miller et al. Only the abstract is used here.\n\nWhat the source observes: Auto-Diagnose, an LLM tool integrated into Critique (Google’s internal code review system), summarizes integration-test failure logs to help developers find the root cause. A manual evaluation on 71 real-world failures found 90.14% root-cause accuracy. After Google-wide deployment, the tool was used across 52,635 distinct failing tests. Users rated it “Not helpful” in 5.8% of cases, and it ranked #14 of 370 Critique tools for helpfulness. Interviews reported perceived usefulness.\n\nWhat the authors argue: LLMs are “highly successful” at this diagnosis task, and accuracy is “a critical factor” in developer perception and adoption.\n\nWhere it bears:\n- software-development, specifically its task “Use it to investigate before you commit.” That task asks for explanations or candidate fixes checked against the actual repository, with the developer keeping responsibility for the diagnosis. This is deployment-scale evidence that AI-assisted diagnosis on a hard, log-heavy part of repair work is accurate in most sampled cases and rarely judged unhelpful.\n- ai-work-quality-and-rework. The main read counts investigating and repairing errors as part of total effort. The arrival suggests AI can reduce some of that checking-and-repair burden itself, not only first-draft production.\n\nLimits: the outcomes are root-cause accuracy on a 71-failure sample and user ratings, not measured diagnosis time, total human effort at fixed quality, or defects after handoff. It therefore meets no signpost or revision trigger. Roughly one in ten sampled diagnoses was wrong, which reinforces the guide’s advice to verify rather than accept the suggestion. Ratings and rank among internal tools are perception measures.\n\nOur reading of context: this is an in-house team evaluating its own deployed tool, which ordinary product-evaluation practice explains. That is a reason to want an independent effort measure, not a reason to discount the accuracy figure.\n\nRival reading: helpfulness ratings may reflect convenience without saving net time, if developers re-verify every diagnosis. A comparison of time-to-root-cause and later fix success with and without the tool would separate the two accounts.","targetIds":["software-development","ai-work-quality-and-rework"],"spans":[{"sourceId":"a3328b3786e180a8fabbe945847cddaeea43d1a4d24a47fee6d4ec7bbad38f66","excerpt":"Developers complain about these difficulties consistently and report spending substantially more time diagnosing integration test failures compared to unit test failures."},{"sourceId":"a3328b3786e180a8fabbe945847cddaeea43d1a4d24a47fee6d4ec7bbad38f66","excerpt":"Auto-Diagnose analyzes failure logs, produces concise summaries with the most relevant log lines, and is integrated into Critique, Google's internal code review system, providing contextual and in-time assistance."},{"sourceId":"a3328b3786e180a8fabbe945847cddaeea43d1a4d24a47fee6d4ec7bbad38f66","excerpt":"A manual evaluation conducted on 71 real-world failures demonstrated 90.14% accuracy in diagnosing the root cause."},{"sourceId":"a3328b3786e180a8fabbe945847cddaeea43d1a4d24a47fee6d4ec7bbad38f66","excerpt":"User feedback indicated that the tool was deemed \"Not helpful\" in only 5.8% of cases, and it was ranked #14 in helpfulness among 370 tools that post findings in Critique."},{"sourceId":"a3328b3786e180a8fabbe945847cddaeea43d1a4d24a47fee6d4ec7bbad38f66","excerpt":"with the tool's accuracy remaining a critical factor in shaping developer perception and adoption."}],"mechanism":null,"assumptions":["Root-cause accuracy on 71 manually evaluated failures and user helpfulness ratings are partial proxies; they do not measure diagnosis time, total human effort at fixed quality or later defects.","The abstract does not describe how the 71 failures were sampled or how accuracy was judged; nothing is inferred from omitted text.","Google’s internal tooling, logs and review culture may not transfer to other employers or codebases."],"intermediateSourceIds":[],"familyId":"text:9d445b9b73bab7c912a9a9c29b6372bf3912c73a4af19a4a9a0c6424e586f806","duplicateOf":null},"references":[{"id":"a3328b3786e180a8fabbe945847cddaeea43d1a4d24a47fee6d4ec7bbad38f66","title":"LLM-Based Automated Diagnosis Of Integration Test Failures At Google","url":"https://arxiv.org/abs/2604.12108"}]}],"revisionUrl":"/guide-maintenance/ai-work-quality-and-rework/history.json?revision=57cdcc2b517a602ecb395e140296d06308bd7c8a11e9e2a674ed538f781a14c6","document":{"questionId":"ai-work-quality-and-rework","questionRead":{"gapLabel":"AI will save effort on work people can readily check, even after they test and correct it","basis":"In 2027–2028, we expect AI to reduce the total effort needed for repeatable digital work when people can readily tell whether the result is correct and useful.\n\nA developer still needs to test AI-generated code against the task’s requirements; a support agent needs to confirm that an AI-suggested answer actually addresses the customer’s problem. Where errors are difficult to detect, investigating and repairing them can consume the time saved by having AI produce the first version. These examples describe the checking people must do; they do not imply that every code change or customer problem is easy to verify.\n\nWe favor savings in the stated scope because the customer-support study found more issues resolved per hour, showing improvement in completed work. In METR’s early-2025 developer experiment, experienced developers working in familiar open-source repositories took 19% longer to complete issues when allowed to use AI. METR’s February 2026 update judges that developers likely benefited more from early-2026 tools, although selection problems prevent a reliable estimate of the change.\n\nDuring 2027–2028, we expect teams to spread these gains by reusing requirements and checking routines across recurring tasks, rather than starting the review process afresh each time.","alternatives":["Even in the defined target workflows, selection and omitted review costs explain most apparent gains; lower effort does not survive a complete comparison.","Models become better at checking their own output, reducing the review burden even on complex work."],"assumptions":["Before deployment, the target work is repeatable and digital, has a clear acceptance test, and can be checked affordably. Quality standards stay fixed and delayed defects count.","Teams can select or redesign tasks instead of forcing every task through the same AI process."],"revisionTriggers":["Comparable later studies measure work completed to the required standard, total human effort and defects after handoff, with credible treatment of selection.","The gain disappears when review and downstream repair are counted, or persists when they are included."],"implication":"The prize is more room for difficult, worthwhile work. Compare finished work of the same quality with the full human effort it took, then use the capacity you recover to improve the service, tackle a neglected problem or give people time back.","estimate":{"status":"editorial-outlook","horizon":{"start":"2027","end":"2028","reason":"Two years for gains already observed in some assisted tasks to spread through workflow redesign. The forecast is about suitable workflows, with hard-to-verify work expected to lag; it is not a uniform productivity percentage."}},"sourceRefs":[{"sourceId":"metr-productivity-2025","spanIds":["setting","completion-time"],"role":"context"},{"sourceId":"metr-productivity-update-2026","spanIds":["selection"],"role":"context"},{"sourceId":"metr-worker-survey-2026","spanIds":["self-report"],"role":"context"},{"sourceId":"anthropic-workforce-2026","spanIds":["anth-bottleneck"],"role":"context"},{"sourceId":"brynjolfsson-ai-work-v2","spanIds":["resolved-work"],"role":"supports"},{"sourceId":"metr-productivity-update-2026","spanIds":["direction-of-selection"],"role":"supports"}],"positionRefs":["metr-productivity-researchers","metr-followup-method","metr-worker-value","customer-support-researchers"],"connectionRefs":["quality-to-reliability","quality-to-returns"],"gapRefs":[{"id":"accepted-work-transfer","reason":"Compare total effort at the same required quality in workflows selected before deployment, including later checking and repair."}]},"questionPaths":[{"id":"quality-useful-work-expands","questionId":"ai-work-quality-and-rework","outcome":"Checkable digital work takes less effort to finish","scope":"Teams lower total effort in repeatable digital workflows with clear acceptance tests and affordable verification. They define those conditions before deployment, then count the work all the way through handoff and repair.","conditions":["The target workflow is repeatable and digital, with a clear acceptance test and affordable verification established before the comparison.","They count verification and later defects, then concentrate use where the full result improves."],"sourceRefs":[{"sourceId":"anthropic-workforce-2026","spanIds":["anth-bottleneck"],"role":"context"},{"sourceId":"metr-worker-survey-2026","spanIds":["self-report"],"role":"context"},{"sourceId":"metr-productivity-2025","spanIds":["setting","completion-time"],"role":"challenges"},{"sourceId":"brynjolfsson-ai-work-v2","spanIds":["resolved-work"],"role":"supports"},{"sourceId":"metr-productivity-update-2026","spanIds":["direction-of-selection"],"role":"supports"}],"positionRefs":["anthropic-work-organization","metr-worker-value","customer-support-researchers"],"horizon":{"start":"2027","end":"2028","reason":"Existing assisted-work gains and improving developer experience favor wider gains over two years, with task selection and redesign driving the spread."},"weight":"base-case","signposts":["Comparable task cohorts show lower total effort at stable quality.","Benefits persist after the novelty of the rollout and include downstream repair."]},{"id":"quality-review-absorbs-gains","questionId":"ai-work-quality-and-rework","outcome":"Even promising workflows take more effort than they save","scope":"The gain disappears even in repeatable digital work selected for clear outcomes and affordable checking. Finding hidden mistakes and fitting the output into the larger job takes more time and effort than expected; faster drafts fail to make finished work easier.","conditions":["Verification and repair cost more than the team expected before rollout.","Selection of enthusiastic users or favorable tasks explains apparent gains that do not survive comparable full-work measurements."],"sourceRefs":[{"sourceId":"metr-productivity-2025","spanIds":["setting","completion-time"],"role":"supports"},{"sourceId":"metr-productivity-update-2026","spanIds":["selection"],"role":"limits"}],"positionRefs":["metr-productivity-researchers","metr-followup-method"],"horizon":{"start":"2027","end":"2028","reason":"This remains a live alternative throughout the adoption window, especially where correctness is expensive to establish."},"weight":"alternative","signposts":["Blind quality review finds more defects in apparently faster work.","Time saved at generation returns as integration, correction or customer support."]}],"dimension":{"id":"software-development","label":"Software development","question":"Which parts of coding are worth handing over today?","verdict":"Developers should use AI regularly for changes they can understand and test, while treating the time spent reviewing and repairing code as part of the job.","who":"Developers who can evaluate the code and its behavior, and learners working with someone who can. Being able to generate a change is not yet being able to maintain it.","tasks":[{"title":"Give it a change with a clear purpose","how":"Specify the behavior you need, the relevant code and how you will test success. Review the proposed change, run it against those requirements and keep the change small enough to understand.","sourceIds":["metr-productivity-2025","metr-productivity-update-2026"]},{"title":"Use it to investigate before you commit","how":"Ask for explanations, candidate fixes or tests, then check them against the actual repository. This lets AI reduce exploration work while you keep responsibility for the diagnosis.","sourceIds":["metr-productivity-2025","metr-productivity-update-2026"]}],"limits":[{"title":"Familiar work may already be faster without it","why":"In METR’s early-2025 experiment, experienced developers working in familiar repositories took 19% longer with AI access. That result is a reason to compare total completion time, not assume a tool always earns its place.","sourceIds":["metr-productivity-2025"]},{"title":"Later improvement is not a measured universal speedup","why":"METR’s February 2026 update judges that developers likely benefited more from newer tools, but selection and time measurement prevent a reliable estimate of the change. Retest the work you actually do.","sourceIds":["metr-productivity-update-2026"]}],"practice":"Keep AI in the parts of development where completed changes reach the required quality with less total effort. Count understanding, testing, integration and later repair alongside generation; remove assistance from a task when it adds more work than it saves.","sourceIds":["metr-productivity-2025","metr-productivity-update-2026","anthropic-workforce-2026","context-chatgpt-source","context-ai-act-source","context-claude-code-source"],"paths":[{"id":"software-whole-tasks","title":"More whole tasks, with review at the finish","summary":"Developers delegate increasingly complete, well-specified changes and concentrate on deciding what to build and whether it works.","mechanism":"Reusable requirements and tests make it easier to judge the result without directing every step. Better tools help only when understanding and repair do not consume the time they save.","horizon":{"start":"2027","end":"2028","label":"During 2027–2028","reason":"EM expects wider delegation during 2027–2028 as improving tools and reusable requirements and tests make complete changes easier to assess."},"conditions":["The change has clear behavior to test and failures can be recovered from.","Developers retain enough understanding to maintain the result."],"implications":["A developer can attempt more useful changes without supervising every keystroke.","Time saved can go into architecture, neglected maintenance or talking to the people who use the software."],"signpost":"Comparable changes require less total human effort while defects after release do not increase.","sourceIds":["metr-productivity-2025","metr-productivity-update-2026","anthropic-workforce-2026"],"kind":"favored"},{"id":"software-reach","title":"Small teams attempt software they could not previously build","summary":"A capable team uses AI to explore more designs, maintain neglected tools or serve a group that could not justify a large development effort.","mechanism":"When some implementation work becomes cheaper, a project that was too expensive can become worth attempting. The opportunity depends on the team being able to test, operate and improve what it creates.","horizon":{"start":null,"end":null,"label":"A future people can shape","reason":"The opportunity opens where reduced implementation effort makes a previously uneconomic project viable. Its timing depends on the team’s ability to operate and maintain what it builds."},"conditions":["The intended users have a real unmet need.","The team budgets time for maintenance, testing and changes after the first working version."],"implications":["More useful tools can exist for smaller communities and organizations.","Developers can spend the recovered capacity on ambitious work rather than only increasing task throughput."],"signpost":"New projects remain useful and maintainable after their initial launch.","sourceIds":["metr-productivity-update-2026","anthropic-workforce-2026"],"kind":"opportunity"},{"id":"software-repair","title":"Code production outruns understanding","summary":"Teams generate changes faster than they can understand, integrate and repair them, so apparent speed creates a growing maintenance burden.","mechanism":"A generated change can move effort from writing code to establishing what it does. If that understanding never catches up, later defects and dependencies bring the work back.","horizon":{"start":"2027","end":"2028","label":"During 2027–2028","reason":"During the 2027–2028 adoption window, teams can expand generated code faster than they improve testing and maintenance. The earlier slowdown makes total completion effort an important test."},"conditions":["Teams judge progress by generated changes rather than working software.","Testing misses important behavior, or maintainers cannot explain the new code."],"implications":["The team spends its next cycle repairing the last one.","People may become responsible for systems they have less time to understand."],"signpost":"Generation accelerates while integration time, failed changes or later repair grow.","sourceIds":["metr-productivity-2025","metr-productivity-update-2026","anthropic-workforce-2026"],"kind":"risk"}],"contextSourceIds":["context-chatgpt-source","context-ai-act-source","context-claude-code-source"]},"foundation":{"editionId":"perspectives-scope-correction-2026-11-14","sources":[{"id":"anthropic-workforce-2026","title":"How employment is changing at Anthropic","url":"https://www.anthropic.com/research/employment-at-anthropic","publishedAt":"2026-10-16","dateBasis":"Publisher publication date, checked on original page.","access":{"status":"public-page","note":"Original publisher page and exact retained source reviewed. Source retention does not establish every downstream reuse or training right.","checkedAt":"2026-10-29"},"provenance":{"kind":"stream-snapshot","workId":"18f3c3ca31491462a8ab52f797ebadd2a7087803317e69a5152e2a0c4cfbed65","snapshotId":"a633ce13b2d661ad9c008ac99d3357f15a80c4bcd91009290d11712c2f565793","textHash":"8ea3533aa08b1f63c143b0a383911da93e88f464edd2d8e61edebf2a97a42232","reason":"Exact retained Stream work/snapshot and SHA256 verified against the 29 October SELECT-only source packet. That lookup found no joined public Stream or bridge reading revision; no reading identity is invented."},"spans":[{"id":"anth-junior","text":"The proportion of junior technical hires fell even faster, from 27% to 9%.","locator":"Retained extracted text, character offsets [4420, 4494). Context (EM paraphrase): Anthropic’s October 2026 report describes its own workforce; junior employees’ share fell from early 2025, and technical hiring still grew in absolute terms. This is a changing hiring mix, not a count of jobs lost."},{"id":"anth-bottleneck","text":"This acceleration makes human attention the bottleneck for agent output.","locator":"Retained extracted text, character offsets [767, 839). Context (EM paraphrase): The authors report rising internal output alongside agent use and interpret human attention as a constraint. Adoption is not randomized; the report does not establish a general causal productivity effect."}],"citation":{"label":"Anthropic workforce report · October 2026","aliases":["Anthropic’s workforce report","Anthropic workforce report","Anthropic’s hiring mix"],"context":"An internal study of employment and agent use at Anthropic. It describes a changing hiring mix and demands on human attention at one growing AI company.","spanIds":["anth-junior","anth-bottleneck"]},"readingRefs":[]},{"id":"metr-productivity-2025","title":"Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity","url":"https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/","publishedAt":"2025-07-10","dateBasis":"Publication date displayed by the original publisher.","access":{"status":"public-page","note":"Original publisher page and exact retained source reviewed. Source retention does not establish every downstream reuse or training right.","checkedAt":"2026-10-29"},"provenance":{"kind":"stream-snapshot","workId":"eeef7ca24568c0fe592aa98201faccea099ec6507dbd9476c8bea56310f28743","snapshotId":"1b377f2c5f76b4960227c32060fef0433519ce76239947c7e836402cb9a945ac","textHash":"78a7743c1dda83f49b4d646c134243e5e6c38bed0eb7aa33d124a37e3971f7df","reason":"Exact retained Stream work/snapshot and SHA256 verified against the 29 October SELECT-only source packet. That lookup found no joined public Stream or bridge reading revision; no reading identity is invented."},"spans":[{"id":"setting","text":"16 experienced developers","locator":"Retained extracted text, character offsets [1527, 1552). Context (EM paraphrase): Methods: developers worked on 246 real issues in familiar open-source repositories using early-2025 tools. This population does not represent all developers or later systems."},{"id":"completion-time","text":"they take 19% longer to complete issues","locator":"Retained extracted text, character offsets [2468, 2507). Context (EM paraphrase): Core Result, first paragraph; interpret with Methods and the immediately following limitations table. Reported result for the 2025 study, not a current universal estimate."}],"citation":{"label":"METR developer experiment · July 2025","aliases":["METR’s early-2025 developer experiment","METR’s 2025 developer study","early-2025 METR experiment"],"context":"A study of experienced developers using early-2025 tools on familiar open-source work. The measured completion-time result belongs to that setting and period.","spanIds":["setting","completion-time"]},"readingRefs":[]},{"id":"metr-productivity-update-2026","title":"We are Changing our Developer Productivity Experiment Design","url":"https://metr.org/blog/2026-02-24-uplift-update/","publishedAt":"2026-02-24","dateBasis":"Publication date displayed by the original publisher.","access":{"status":"public-page","note":"Original publisher page and exact retained source reviewed. Source retention does not establish every downstream reuse or training right.","checkedAt":"2026-10-29"},"provenance":{"kind":"stream-snapshot","workId":"54d658ff870e4bbb0638a847bac138a3b543d42ac48e761c915c570500f1cc5c","snapshotId":"8fd5f006c0dc55d1e20448712d3f4b177c53cd92219b9aa3ed71b70d0ed13eca","textHash":"81b6c9b7d710e11ed96cf12de5f307e8551454cfdcc9f6c66c84f0a657b9adfb","reason":"Exact retained Stream work/snapshot and SHA256 verified against the 29 October SELECT-only source packet. That lookup found no joined public Stream or bridge reading revision; no reading identity is invented."},"spans":[{"id":"selection","text":"the data from our new experiment gives us an unreliable signal of the current productivity effect of AI tools","locator":"Retained extracted text, character offsets [431, 540). Context (EM paraphrase): Opening discussion, paragraph beginning Unfortunately. Subsequent paragraphs describe participation, task selection and time measurement."},{"id":"direction-of-selection","text":"likely biases downwards","locator":"Retained extracted text, character offsets [715, 738). Context (EM paraphrase): METR says nonparticipation by developers unwilling to work without AI likely understates speedup; the authors believe early-2026 tools speed developers more than early-2025 tools. Lower study pay, task selection and concurrent-agent time accounting limit the size estimate."}],"citation":{"label":"METR developer follow-up · February 2026","aliases":["METR’s February 2026 update","METR’s 2026 follow-up","METR’s later developer evidence"],"context":"METR judges developer benefits likely improved while explaining why participation, task selection and time measurement make the size of the change hard to estimate.","spanIds":["selection","direction-of-selection"]},"readingRefs":[]},{"id":"metr-worker-survey-2026","title":"Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity","url":"https://metr.org/blog/2026-05-11-ai-usage-survey/","publishedAt":"2026-05-11","dateBasis":"Publication date displayed by the original publisher.","access":{"status":"public-page","note":"Original publisher page and exact retained source reviewed. Source retention does not establish every downstream reuse or training right.","checkedAt":"2026-10-29"},"provenance":{"kind":"stream-snapshot","workId":"1f24eee26acc9583eb4649a75f64560f3c33c7db3211c7c6b2997a22eabf6d93","snapshotId":"136edbda94fff696943b30a2b15f34cee5c35b551ad06af769dd6bed2e189ace","textHash":"8b47a32be31a689e627a34baf7b4d2e8998c64dc27e4c9442e0d431b9e502b8f","reason":"Exact retained Stream work/snapshot and SHA256 verified against the 29 October SELECT-only source packet. That lookup found no joined public Stream or bridge reading revision; no reading identity is invented."},"spans":[{"id":"self-report","text":"Participants self-reported a median 1.4–2x change in the value in their work due to AI tools.","locator":"Retained extracted text, character offsets [1308, 1401). Context (EM paraphrase): METR surveyed 349 technical workers in February–April 2026. The range reflects self-reported value under different question formulations, not a confidence interval or observed causal return. METR gives reasons to doubt the magnitude and distinguishes value from speed."}],"citation":{"label":"METR technical-worker survey · May 2026","aliases":["METR’s technical-worker survey","METR’s May 2026 survey"],"context":"A survey of technical workers’ self-reported value from AI tools. It distinguishes value from speed and does not measure an audited financial return.","spanIds":["self-report"]},"readingRefs":[]},{"id":"brynjolfsson-ai-work-v2","title":"Generative AI at Work — arXiv v2","url":"https://arxiv.org/abs/2304.11771v2","publishedAt":"2024-11-06","dateBasis":"Version 2 submission date displayed by arXiv; first submitted 23 April 2023. The versioned abstract is the cited object.","access":{"status":"public-page","note":"Versioned author abstract checked directly. The HTML rendering displayed a conflicting 2026 manuscript date, so this edition cites only the clearly versioned abstract and does not assign that later date to the study.","checkedAt":"2026-10-30"},"provenance":{"kind":"external","workId":null,"snapshotId":null,"textHash":null,"reason":"Original public source read on 30 October 2026. An exact retained Stream identity has not been resolved for this edition."},"spans":[{"id":"resolved-work","text":"by 15\\% on average","locator":"Version 2 abstract, second sentence. EM context: staggered introduction among 5,172 customer-support agents; the outcome is issues resolved per hour, not company profit."},{"id":"novice-gains","text":"Less experienced and lower-skilled workers improve both the speed and quality of their output","locator":"Version 2 abstract, third sentence. EM context: benefits differ by experience; the most experienced workers show small speed gains and small quality declines."}],"citation":{"label":"Brynjolfsson, Li and Raymond · Generative AI at Work","aliases":["the customer-support study","customer-support research","customer-support evidence","customer-support results","customer-support comparison","Generative AI at Work"],"context":"A study of AI assistance for customer-support agents, reporting issues resolved per hour and differences by worker experience. This edition cites the arXiv v2 abstract.","spanIds":["resolved-work","novice-gains"]},"readingRefs":[]}],"positions":[{"id":"metr-productivity-researchers","speaker":"Joel Becker, Nate Rush, Beth Barnes and David Rein","role":"METR authors of the early-2025 controlled developer study","statement":"In their randomized study of 16 experienced open-source developers completing 246 issues, AI access increased completion time by 19%. They explicitly restrict what this says about other developers, settings or later tools.","conditions":["The study used early-2025 tools in developers’ familiar repositories; it is not an estimate for all work or present-day AI."],"sourceRefs":[{"sourceId":"metr-productivity-2025","spanIds":["setting","completion-time"],"role":"supports"}],"statedAt":"2025-07-10","dateBasis":"Publication date of this study report.","attribution":"direct"},{"id":"anthropic-work-organization","speaker":"Nathan Wilmers, Regina Catipon, Maxim Massenkoff and Peter McCrory","role":"Anthropic Institute researchers reporting on their employer","statement":"The report describes increased output alongside rising demands on human attention as AI use spreads internally.","conditions":["Observations at one AI company; not a controlled estimate of the causal effect or of other employers’ results."],"sourceRefs":[{"sourceId":"anthropic-workforce-2026","spanIds":["anth-bottleneck"],"role":"supports"}],"statedAt":"2026-10-16","dateBasis":"Dated authored publication; this is not the date of every underlying observation or interview.","attribution":"direct"},{"id":"metr-followup-method","speaker":"Joel Becker, Nate Rush, Tom Cunningham, David Rein and Khalid Mahamud","role":"METR authors of the follow-up methods update","statement":"METR judges that developers likely benefited more from AI in early 2026 than in early 2025, while warning that participation, task selection and time measurement make the size of that improvement hard to estimate.","conditions":["This is a later team’s assessment of a changed experiment, not the same measurement repeated cleanly."],"sourceRefs":[{"sourceId":"metr-productivity-update-2026","spanIds":["selection"],"role":"supports"},{"sourceId":"metr-productivity-update-2026","spanIds":["direction-of-selection"],"role":"supports"}],"statedAt":"2026-02-24","dateBasis":"Publication date of the authored source.","attribution":"direct"},{"id":"metr-worker-value","speaker":"Joel Becker","role":"METR author reporting technical-worker self-assessments","statement":"The survey reports substantial perceived value gains, while distinguishing value from task speed and cautioning that respondents can misestimate what AI contributes.","conditions":["Self-report in a technical-worker sample, not audited customer economics or a measured effect on national productivity."],"sourceRefs":[{"sourceId":"metr-worker-survey-2026","spanIds":["self-report"],"role":"supports"}],"statedAt":"2026-05-11","dateBasis":"Publication date of the authored source.","attribution":"direct"},{"id":"customer-support-researchers","speaker":"Erik Brynjolfsson, Danielle Li and Lindsey Raymond","role":"Researchers studying AI assistance in customer support","statement":"The researchers find higher resolved issues per hour after AI assistance, with the largest benefits for less experienced workers.","conditions":["One employer and an assisted workflow; the study does not estimate aggregate employment or full deployment returns."],"sourceRefs":[{"sourceId":"brynjolfsson-ai-work-v2","spanIds":["resolved-work","novice-gains"],"role":"supports"}],"statedAt":"2024-11-06","dateBasis":"Date of the cited version or source update.","attribution":"direct"}]},"relatedDimensions":[{"id":"customer-support","label":"Customer support","question":"Can AI help people resolve more problems without making service worse?","verdict":"Support teams should use AI regularly on recurring questions, with agents responsible for whether the answer actually resolves the customer’s problem.","who":"Agents with access to reliable product and policy information, especially people learning the role. Experienced agents should keep the assistance that improves their work, rather than assume every suggestion is an improvement.","tasks":[{"title":"Prepare a useful answer","how":"Give the assistant the customer’s issue and the relevant product or policy material. Ask for a proposed response and its supporting passages; confirm the details before promising an outcome.","sourceIds":["brynjolfsson-ai-work-v2"]},{"title":"Help a new agent understand the case","how":"Use a suggested response as a starting point for discussing why a solution fits this customer. A supervisor can review the reasoning and the final resolution together.","sourceIds":["brynjolfsson-ai-work-v2"]}],"limits":[{"title":"A quick reply can leave the problem unsolved","why":"The study’s useful outcome was issues resolved per hour. Count repeat contacts and unresolved cases as well as response speed, so faster drafting does not hide work shifted back to the customer.","sourceIds":["brynjolfsson-ai-work-v2"]},{"title":"Experts do not benefit in the same way as beginners","why":"The cited support study found larger gains for less experienced workers; the most experienced showed small speed gains and small quality declines. Keep experienced judgment in the process.","sourceIds":["brynjolfsson-ai-work-v2"]}],"practice":"Choose a recurring kind of customer problem, give agents dependable reference material, and compare the quality of resolution as well as the effort it takes. Use the capacity recovered to improve service, not merely to send more replies.","sourceIds":["brynjolfsson-ai-work-v2","anthropic-workforce-2026","stanford-canaries-2026","context-chatgpt-source","context-ai-act-source"],"paths":[{"id":"support-more-resolution","title":"More problems resolved with the same team","summary":"AI becomes a routine source of suggested answers, leaving agents more room for unusual cases and customers who need sustained attention.","mechanism":"When reliable reference material and human judgment turn suggestions into correct resolutions, less time is spent assembling each routine answer. The team can use that capacity for work its old queue left waiting.","horizon":{"start":"2027","end":"2027","label":"During 2027","reason":"EM expects wider routine assistance during 2027 because a real support deployment already demonstrates gains in completed work. Teams can extend that approach where reference material and review are dependable."},"conditions":["The assistant can use current product and policy material.","Agents can reject a suggestion and escalate a case; the team tracks whether customers get a resolution."],"implications":["A smaller queue can mean more attention per difficult case.","Managers can choose better service or greater volume; the tool does not make that choice for them."],"signpost":"Resolved cases per agent improve without more repeat contacts or lower service quality.","sourceIds":["brynjolfsson-ai-work-v2","anthropic-workforce-2026"],"kind":"favored"},{"id":"support-learning","title":"The assistant helps more people become capable agents","summary":"A team uses AI assistance to widen access to skilled support work, helping new hires learn why an answer works as well as what to say.","mechanism":"The study shows novices can benefit while working with assistance. Turning that advantage into lasting competence requires employers to pair useful suggestions with explanation, practice and growing responsibility.","horizon":{"start":null,"end":null,"label":"A future people can shape","reason":"The opportunity depends on employers turning assistance into explanation, practice and growing responsibility. Better performance with a tool becomes lasting competence only if people also learn the work."},"conditions":["Employers keep paid entry roles and time for explanation.","Supervisors check whether new agents can handle increasingly unfamiliar cases, including without a ready-made suggestion."],"implications":["People can enter skilled work with more help at the beginning.","The team develops experienced colleagues instead of only increasing immediate output."],"signpost":"New hires progress to independent responsibility while customer outcomes hold up.","sourceIds":["brynjolfsson-ai-work-v2","stanford-canaries-2026"],"kind":"opportunity"},{"id":"support-volume","title":"Faster replies become poorer service","summary":"A team uses the extra capacity to raise targets while customers still need repeated contact to get their actual problem resolved.","mechanism":"If managers reward response volume and remove the time to check an answer, quick suggestions can pass unresolved work to customers and the next agent.","horizon":{"start":"2027","end":"2028","label":"During 2027–2028","reason":"This risk belongs to the 2027–2028 rollout window: faster production can outrun service checks as teams raise targets."},"conditions":["Response targets rise faster than the team’s ability to check solutions.","Repeat contacts and errors are excluded from the success measure."],"implications":["Customers spend more effort navigating apparently faster service.","Agents inherit frustration and correction work that the first response did not finish."],"signpost":"Reply speed improves while repeat contacts, escalations or unresolved complaints rise.","sourceIds":["brynjolfsson-ai-work-v2","anthropic-workforce-2026"],"kind":"risk"}],"contextSourceIds":["context-chatgpt-source","context-ai-act-source"]},{"id":"writing-analysis","label":"Writing and analysis","question":"Can AI help you think and communicate without doing the deciding for you?","verdict":"People who write for work should use AI regularly to develop and revise drafts. Analysts should also use it to explore alternatives, while checking the evidence and calculations that determine the conclusion.","who":"Writers, researchers, analysts and managers who can supply the relevant material and judge whether the output represents it accurately. The strongest case is help with the work of expression; a fluent answer does not settle whether an analysis is right.","tasks":[{"title":"Turn your material into a draft you can improve","how":"Provide the facts, audience and purpose. Ask for an outline or a draft, then decide what is missing, unsupported or unlike what you mean. Keep the useful structure and rewrite the argument where needed.","sourceIds":["noy-zhang-writing-2023"]},{"title":"Develop alternatives before choosing a conclusion","how":"Ask for competing explanations, possible objections or different ways to present the same evidence. Check the claims against the original material and perform consequential calculations in a tool where you can inspect them.","sourceIds":["dellacqua-jagged-frontier-2026"]}],"limits":[{"title":"Writing quality is not factual reliability","why":"The 2023 writing experiment found faster completion and higher evaluator-rated quality on short professional tasks. Those tasks did not test all the factual precision and organizational context that real assignments can require.","sourceIds":["noy-zhang-writing-2023"]},{"title":"Persuasive analysis can be wrong","why":"In the consulting experiment using 2023 GPT-4, AI helped on many tasks but reduced correctness on a complex managerial case. Success on drafting does not establish competence at the next analytical step.","sourceIds":["dellacqua-jagged-frontier-2026"]}],"practice":"Use AI to help you articulate, compare and challenge ideas. Keep the original evidence close enough to inspect, make your own argument, and spend part of the time saved asking whether the conclusion deserves to survive.","sourceIds":["noy-zhang-writing-2023","dellacqua-jagged-frontier-2026","context-chatgpt-source","context-ai-act-source"],"paths":[{"id":"writing-more-judgment","title":"Less effort on first drafts, more on the argument","summary":"AI becomes routine for outlines, drafts and revision, while the distinctive work shifts toward choosing the evidence and deciding what is worth saying.","mechanism":"The writing evidence shows that generating and shaping text can take less effort. When people preserve time to evaluate the substance, that saving can improve the finished communication instead of merely increasing its volume.","horizon":{"start":"2027","end":"2028","label":"During 2027–2028","reason":"EM expects gains in professional drafting to spread during 2027–2028 as teams reuse briefs and review practices. The dated studies establish an opportunity that organizations can turn into repeatable work."},"conditions":["Writers provide the purpose and relevant material.","Review checks the argument and factual claims, not just tone and grammar."],"implications":["People can communicate ideas they previously struggled to put into words.","The time recovered can go into investigation, conversation or a stronger second draft."],"signpost":"Completed writing takes less total effort while readers find it accurate, useful and clear.","sourceIds":["noy-zhang-writing-2023","dellacqua-jagged-frontier-2026"],"kind":"favored"},{"id":"writing-more-voices","title":"More people get to develop and share an idea","summary":"People use help with expression to contribute proposals, explanations and creative work that would otherwise remain unfinished.","mechanism":"Reducing the effort of drafting can lower one barrier to participation. The benefit becomes meaningful when people retain their own purpose, experience and voice and have somewhere worthwhile to contribute.","horizon":{"start":null,"end":null,"label":"A future people can shape","reason":"The opportunity depends on writing assistance becoming useful to more people and on organizations making room for their contributions."},"conditions":["Tools help people express their own material rather than replace it with a generic position.","Teams and communities make room to read, question and develop the contributions."],"implications":["People with useful experience can spend more energy on their idea and less on getting a first version onto the page.","An organization can hear from more of the people who understand its work."],"signpost":"A wider range of contributors produce useful work whose reasoning and voice remain their own.","sourceIds":["noy-zhang-writing-2023","dellacqua-jagged-frontier-2026"],"kind":"opportunity"},{"id":"writing-polished-error","title":"Polished output gets ahead of the evidence","summary":"People produce convincing recommendations more quickly while spending less time checking whether the central claim is true.","mechanism":"If a well-structured answer is mistaken for a sound analysis, fluency makes a weak conclusion easier to circulate. The consulting experiment shows why quality of presentation and correctness must be examined separately.","horizon":{"start":null,"end":null,"label":"As AI-assisted drafting becomes routine","reason":"The risk arises whenever confidence in a polished draft outruns the checking of its evidence."},"conditions":["Readers use polish as a substitute for inspecting evidence.","Writers cannot reconstruct the calculation or explain why the recommendation follows."],"implications":["More persuasive documents can lead to worse decisions.","Reviewers inherit the harder job of locating errors inside apparently complete work."],"signpost":"The writing becomes clearer while independent checks find more unsupported claims or incorrect conclusions.","sourceIds":["dellacqua-jagged-frontier-2026","noy-zhang-writing-2023"],"kind":"risk"}],"contextSourceIds":["context-chatgpt-source","context-ai-act-source"]}]}}