# Google engineers’ randomized trial finds AI shortened time on an enterprise task

[Canonical reading](https://escapemeridian.com/readings/google-engineers-randomized-trial-finds-ai-shortened-time-on-an-09b29f95)

The authors report that AI assistance significantly shortened the time 96 full-time Google software engineers spent on a complex, enterprise-grade task. Their best estimate of the reduction is about 21%, controlling for factors known to influence time on task, though they say the confidence interval is large. The result concerns time spent on this task, not every measure of developer productivity: earlier studies have examined outcomes ranging from perceived productivity to speed on task and developer throughput. The authors also report that developers who spent more hours per day on code-related activities were faster with AI. They caution that the estimate may not apply more broadly: the study used internal Google tooling in summer 2024, and they cannot assume its effect will translate across tools or over time. They invite further research at the ecosystem level and across multiple suites of AI-enhanced tools. The study offers evidence about one enterprise-grade task and tool setting, while leaving broader effects open.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv · Article

Original source date: Oct 16, 2024 (2024-10-16).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-20T13:58:42.185Z. First observed: 2026-11-20T13:57:46.872Z. Retrieved: 2026-11-20T13:57:54.444Z.

- [How much does AI impact development speed? An enterprise-based randomized controlled trial](https://arxiv.org/abs/2410.12944)

## Limits

The study estimates time spent on a complex, enterprise-grade task; its authors caution that the effect size may not apply more broadly or translate across tools and over time. The abstract reports a large confidence interval and does not provide its bounds.


## Open question

How does AI’s impact on developer time on task vary at the ecosystem level and across multiple suites of AI-enhanced tools?


## Source-stated

The authors report that AI significantly shortened the time 96 full-time Google software engineers spent on a complex, enterprise-grade task.

> Our randomized controlled trial with 96 full-time Google software engineers contributes to this literature by sharing an estimate of the impact of three AI features on the time developers spent on a complex, enterprise-grade task. We found that AI significantly shortened the time developers spent on

- [Supporting source](https://arxiv.org/abs/2410.12944)

## Source-stated

The authors’ best estimate of the effect is about 21%, with a large confidence interval.

> Our best estimate of the size of this effect, controlling for factors known to influence developer time on task, stands at about 21\\%, although our confidence interval is large.

- [Supporting source](https://arxiv.org/abs/2410.12944)

## Source-stated

The authors report that developers who spend more hours on code-related activities per day were faster with AI.

> We also found an interesting effect whereby developers who spend more hours on code-related activities per day were faster with AI.

- [Supporting source](https://arxiv.org/abs/2410.12944)

## Source-stated

The authors caution that the study’s effect size may not generalize across settings, tools, or time, and call for further ecosystem-level research across AI-enhanced tool suites.

> we cannot assume that the effect size obtained in our lab study will necessarily apply more broadly, or that the effect of AI found using internal Google tooling in the summer of 2024 will translate across tools and over time.

- [Supporting source](https://arxiv.org/abs/2410.12944)

## Source and contributor context

Paradis, Elise, Grey, Kate, Madison, Quinn, Nam, Daye, Macvean, Andrew, Meimand, Vahid, Zhang, Nan, Ferrari-Church, Ben, Chandra, Satish · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Randomized controlled trial

A study that randomly assigns people to groups so that differences in outcomes can be traced to the treatment.

- [Definition reference](https://en.wikipedia.org/wiki/Randomized_controlled_trial)

## Confidence interval

A range of values consistent with the data at a stated level of confidence.

- [Definition reference](https://en.wikipedia.org/wiki/Confidence_interval)

## TIME

TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.

- [Explore this context](/entities/time)
- [Reference](https://time.com/)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: 63c2eb04-6e97-4e52-8372-9668ed6245a9.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-20T13:58:42.185Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: d2fca925089dd3170745e088b3db2c533896957c0827f5a80ee81b3aeb9e5e4a. Source text hash: dc32247ea63e69de4ada0f38f7b7a300652e7370de83c64ff309eca970a49d0b.


## Where this evidence matters

- [Does AI reduce the effort needed to finish work well?](/questions/ai-work-quality-and-rework#our-read) · direct · supports
  The arrival is the arXiv v3 abstract (v1 submitted 16 October 2024; v3 revised 11 November 2024) of Paradis et al.'s randomized controlled trial. Its population (software engineers), task (development work) and outcome type (comparative time on task) fall within the guide's software scope. It is a new evidence family. It is not a report of METR, Peng et al., Anthropic or the Narayanan and Kapoor essay.

What the source observes: in a randomized lab trial, 96 full-time Google engineers were given three AI features on one "complex, enterprise-grade task". AI "significantly shortened" their time on task. The authors' best estimate is about 21%, with a large confidence interval. Developers who spend more daily hours on code were faster with AI.

What the authors argue: the effect size need not apply more broadly. They also say results from internal Google tooling in summer 2024 may not translate across tools or over time.

Where it bears:
- The main question's claim that AI can lower effort on defined, checkable work.
- quality-useful-work-expands.
- software-whole-tasks, which concerns delegating well-specified changes.
- quality-review-absorbs-gains:condition:1, the rival that selection of enthusiastic users or favorable tasks explains apparent gains. Randomization addresses participant selection. The setting is also a complex enterprise task rather than a small greenfield task, so this weakens the rival somewhat relative to the Peng et al. Copilot result.

Limits: the outcome is time on task in a lab. The abstract states no quality standard or acceptance test, and it does not count later review, integration or defects after handoff. It therefore does not meet the signpost of lower total effort at stable quality or the revision trigger on defects after handoff.

It does not contradict METR's early-2025 19% slowdown. The settings differ: a researcher-set lab task with company tooling versus experienced developers working real issues in their own familiar repositories. That difference fits the guide's existing scope distinction rather than resolving it.
  [Read the revision from 2026-11-20](/guide-maintenance/ai-work-quality-and-rework/history.json?revision=57cdcc2b517a602ecb395e140296d06308bd7c8a11e9e2a674ed538f781a14c6)

- [Checkable digital work takes less effort to finish](/questions/ai-work-quality-and-rework#path-quality-useful-work-expands) · direct · supports
  The arrival is the arXiv v3 abstract (v1 submitted 16 October 2024; v3 revised 11 November 2024) of Paradis et al.'s randomized controlled trial. Its population (software engineers), task (development work) and outcome type (comparative time on task) fall within the guide's software scope. It is a new evidence family. It is not a report of METR, Peng et al., Anthropic or the Narayanan and Kapoor essay.

What the source observes: in a randomized lab trial, 96 full-time Google engineers were given three AI features on one "complex, enterprise-grade task". AI "significantly shortened" their time on task. The authors' best estimate is about 21%, with a large confidence interval. Developers who spend more daily hours on code were faster with AI.

What the authors argue: the effect size need not apply more broadly. They also say results from internal Google tooling in summer 2024 may not translate across tools or over time.

Where it bears:
- The main question's claim that AI can lower effort on defined, checkable work.
- quality-useful-work-expands.
- software-whole-tasks, which concerns delegating well-specified changes.
- quality-review-absorbs-gains:condition:1, the rival that selection of enthusiastic users or favorable tasks explains apparent gains. Randomization addresses participant selection. The setting is also a complex enterprise task rather than a small greenfield task, so this weakens the rival somewhat relative to the Peng et al. Copilot result.

Limits: the outcome is time on task in a lab. The abstract states no quality standard or acceptance test, and it does not count later review, integration or defects after handoff. It therefore does not meet the signpost of lower total effort at stable quality or the revision trigger on defects after handoff.

It does not contradict METR's early-2025 19% slowdown. The settings differ: a researcher-set lab task with company tooling versus experienced developers working real issues in their own familiar repositories. That difference fits the guide's existing scope distinction rather than resolving it.
  [Read the revision from 2026-11-20](/guide-maintenance/ai-work-quality-and-rework/history.json?revision=57cdcc2b517a602ecb395e140296d06308bd7c8a11e9e2a674ed538f781a14c6)

- [More whole tasks, with review at the finish](/?work=software-development&path=software-whole-tasks#possibilities) · direct · supports
  The arrival is the arXiv v3 abstract (v1 submitted 16 October 2024; v3 revised 11 November 2024) of Paradis et al.'s randomized controlled trial. Its population (software engineers), task (development work) and outcome type (comparative time on task) fall within the guide's software scope. It is a new evidence family. It is not a report of METR, Peng et al., Anthropic or the Narayanan and Kapoor essay.

What the source observes: in a randomized lab trial, 96 full-time Google engineers were given three AI features on one "complex, enterprise-grade task". AI "significantly shortened" their time on task. The authors' best estimate is about 21%, with a large confidence interval. Developers who spend more daily hours on code were faster with AI.

What the authors argue: the effect size need not apply more broadly. They also say results from internal Google tooling in summer 2024 may not translate across tools or over time.

Where it bears:
- The main question's claim that AI can lower effort on defined, checkable work.
- quality-useful-work-expands.
- software-whole-tasks, which concerns delegating well-specified changes.
- quality-review-absorbs-gains:condition:1, the rival that selection of enthusiastic users or favorable tasks explains apparent gains. Randomization addresses participant selection. The setting is also a complex enterprise task rather than a small greenfield task, so this weakens the rival somewhat relative to the Peng et al. Copilot result.

Limits: the outcome is time on task in a lab. The abstract states no quality standard or acceptance test, and it does not count later review, integration or defects after handoff. It therefore does not meet the signpost of lower total effort at stable quality or the revision trigger on defects after handoff.

It does not contradict METR's early-2025 19% slowdown. The settings differ: a researcher-set lab task with company tooling versus experienced developers working real issues in their own familiar repositories. That difference fits the guide's existing scope distinction rather than resolving it.
  [Read the revision from 2026-11-20](/guide-maintenance/ai-work-quality-and-rework/history.json?revision=57cdcc2b517a602ecb395e140296d06308bd7c8a11e9e2a674ed538f781a14c6)

- [Even promising workflows take more effort than they save](/questions/ai-work-quality-and-rework#path-quality-review-absorbs-gains) · direct · supports
  The arrival is the arXiv v3 abstract (v1 submitted 16 October 2024; v3 revised 11 November 2024) of Paradis et al.'s randomized controlled trial. Its population (software engineers), task (development work) and outcome type (comparative time on task) fall within the guide's software scope. It is a new evidence family. It is not a report of METR, Peng et al., Anthropic or the Narayanan and Kapoor essay.

What the source observes: in a randomized lab trial, 96 full-time Google engineers were given three AI features on one "complex, enterprise-grade task". AI "significantly shortened" their time on task. The authors' best estimate is about 21%, with a large confidence interval. Developers who spend more daily hours on code were faster with AI.

What the authors argue: the effect size need not apply more broadly. They also say results from internal Google tooling in summer 2024 may not translate across tools or over time.

Where it bears:
- The main question's claim that AI can lower effort on defined, checkable work.
- quality-useful-work-expands.
- software-whole-tasks, which concerns delegating well-specified changes.
- quality-review-absorbs-gains:condition:1, the rival that selection of enthusiastic users or favorable tasks explains apparent gains. Randomization addresses participant selection. The setting is also a complex enterprise task rather than a small greenfield task, so this weakens the rival somewhat relative to the Peng et al. Copilot result.

Limits: the outcome is time on task in a lab. The abstract states no quality standard or acceptance test, and it does not count later review, integration or defects after handoff. It therefore does not meet the signpost of lower total effort at stable quality or the revision trigger on defects after handoff.

It does not contradict METR's early-2025 19% slowdown. The settings differ: a researcher-set lab task with company tooling versus experienced developers working real issues in their own familiar repositories. That difference fits the guide's existing scope distinction rather than resolving it.
  [Read the revision from 2026-11-20](/guide-maintenance/ai-work-quality-and-rework/history.json?revision=57cdcc2b517a602ecb395e140296d06308bd7c8a11e9e2a674ed538f781a14c6)

## Revision and source-check history

### Version 1 — 2026-11-20T13:58:42.185Z

The authors report that AI assistance significantly shortened the time 96 full-time Google software engineers spent on a complex, enterprise-grade task. Their best estimate of the reduction is about 21%, controlling for factors known to influence time on task, though they say the confidence interval is large. The result concerns time spent on this task, not every measure of developer productivity: earlier studies have examined outcomes ranging from perceived productivity to speed on task and developer throughput. The authors also report that developers who spent more hours per day on code-related activities were faster with AI. They caution that the estimate may not apply more broadly: the study used internal Google tooling in summer 2024, and they cannot assume its effect will translate across tools or over time. They invite further research at the ecosystem level and across multiple suites of AI-enhanced tools. The study offers evidence about one enterprise-grade task and tool setting, while leaving broader effects open.

## Original source

arXiv · Article

Original source date: Oct 16, 2024 (2024-10-16).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-20T13:58:42.185Z. First observed: 2026-11-20T13:57:46.872Z. Retrieved: 2026-11-20T13:57:54.444Z.

- [How much does AI impact development speed? An enterprise-based randomized controlled trial](https://arxiv.org/abs/2410.12944)

## Limits

The study estimates time spent on a complex, enterprise-grade task; its authors caution that the effect size may not apply more broadly or translate across tools and over time. The abstract reports a large confidence interval and does not provide its bounds.


## Open question

How does AI’s impact on developer time on task vary at the ecosystem level and across multiple suites of AI-enhanced tools?


## Source-stated

The authors report that AI significantly shortened the time 96 full-time Google software engineers spent on a complex, enterprise-grade task.

> Our randomized controlled trial with 96 full-time Google software engineers contributes to this literature by sharing an estimate of the impact of three AI features on the time developers spent on a complex, enterprise-grade task. We found that AI significantly shortened the time developers spent on

- [Supporting source](https://arxiv.org/abs/2410.12944)

## Source-stated

The authors’ best estimate of the effect is about 21%, with a large confidence interval.

> Our best estimate of the size of this effect, controlling for factors known to influence developer time on task, stands at about 21\\%, although our confidence interval is large.

- [Supporting source](https://arxiv.org/abs/2410.12944)

## Source-stated

The authors report that developers who spend more hours on code-related activities per day were faster with AI.

> We also found an interesting effect whereby developers who spend more hours on code-related activities per day were faster with AI.

- [Supporting source](https://arxiv.org/abs/2410.12944)

## Source-stated

The authors caution that the study’s effect size may not generalize across settings, tools, or time, and call for further ecosystem-level research across AI-enhanced tool suites.

> we cannot assume that the effect size obtained in our lab study will necessarily apply more broadly, or that the effect of AI found using internal Google tooling in the summer of 2024 will translate across tools and over time.

- [Supporting source](https://arxiv.org/abs/2410.12944)

## Source and contributor context

Paradis, Elise, Grey, Kate, Madison, Quinn, Nam, Daye, Macvean, Andrew, Meimand, Vahid, Zhang, Nan, Ferrari-Church, Ben, Chandra, Satish · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Randomized controlled trial

A study that randomly assigns people to groups so that differences in outcomes can be traced to the treatment.

- [Definition reference](https://en.wikipedia.org/wiki/Randomized_controlled_trial)

## Confidence interval

A range of values consistent with the data at a stated level of confidence.

- [Definition reference](https://en.wikipedia.org/wiki/Confidence_interval)

## TIME

TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.

- [Explore this context](/entities/time)
- [Reference](https://time.com/)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: 63c2eb04-6e97-4e52-8372-9668ed6245a9.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-20T13:58:42.185Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: d2fca925089dd3170745e088b3db2c533896957c0827f5a80ee81b3aeb9e5e4a. Source text hash: dc32247ea63e69de4ada0f38f7b7a300652e7370de83c64ff309eca970a49d0b.


### passed — 2026-11-20T13:58:42.185Z — anthropic/claude-opus-5.5

Publication: 63c2eb04-6e97-4e52-8372-9668ed6245a9

attribution — pass: The reading consistently attributes findings and cautions to the authors of the arXiv paper. It describes the participants as 96 full-time Google software engineers, which matches the source. The title phrase "Google engineers' randomized trial" refers to a trial of Google engineers. The abstract does not state the authors' affiliation, and the reading does not assert one.

> Our randomized controlled trial with 96 full-time Google software engineers

summary — pass: The reading leads with the finding: AI shortened time on task by an estimated 21%, with a large confidence interval. It scopes the result to time on task rather than all productivity measures, and keeps the authors' generalization caveats and call for ecosystem-level research. The question follows from the source. Two minor omissions do not distort meaning: the source calls this a "lab study", and the trial tested three AI features. Adding "lab" would be a small fidelity gain.

> we cannot assume that the effect size obtained in our lab study will necessarily apply more broadly

date — pass: The originalPublished value of 2024-10-16 matches the v1 submission date. Later revisions (v2 on Oct 24, v3 on Nov 11) correctly do not set the date.

> \[Submitted on 16 Oct 2024 (v1), last revised 11 Nov 2024 (this version, v3)\]

support — pass: Each claim is source-stated with an exact supporting excerpt: the significant reduction, the roughly 21% estimate with a large confidence interval, the effect for developers who code more hours per day, and the generalization caveats. The limitation that the abstract gives no interval bounds is accurate. The reading does not invent benefits or costs.

> Our best estimate of the size of this effect, controlling for factors known to influence developer time on task, stands at about 21\\%, although our confidence interval is large.

media — pass: The selected media is the verified arXiv abstract page itself. It is the direct original source and the most relevant link.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/09b29f95e46a8f0bbc508732859a0ecc9370d497ffac4e63252b009e932677fc/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/google-engineers-randomized-trial-finds-ai-shortened-time-on-an-09b29f95.json)
