Peng, Sida, Kalliamvakou, Eirini, Cihon, Peter, Demirer, Mert · Feb 13, 2023
arXiv: The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
Markdown · JSON · Original source
Authors Sida Peng, Eirini Kalliamvakou, Peter Cihon and Mert Demirer report a controlled experiment testing whether GitHub Copilot, described as an AI pair programmer, helped software developers complete a programming task faster. Recruited developers were asked to implement an HTTP server in JavaScript as quickly as possible. The authors say the group with access to Copilot completed the task 55.8% faster than the control group. This result concerns performance on that task in the experiment; it does not by itself establish how much Copilot changes productivity across other kinds of software development. The authors also report heterogenous effects and say these show promise for helping people transition into software development careers. The abstract does not establish whether that promise translates into better career outcomes.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
· Article
Source details
Original source date: Feb 13, 2023 (2023-02-13).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-20T12:27:28.052Z. First observed: 2026-11-20T03:27:16.171Z. Retrieved: 2026-11-20T12:26:50.782Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The reported task was implementing an HTTP server in JavaScript; the abstract does not establish that the result applies to other software development tasks. The abstract does not give details of how the observed heterogeneous effects varied or report outcomes for people transitioning into software development careers.
Open question
How do the observed effects vary across developers, and do they extend to software development tasks beyond implementing an HTTP server in JavaScript?
Source-stated
In the controlled experiment described by the authors, developers with access to GitHub Copilot completed the assigned task faster than the control group.
The treatment group, with access to the AI pair programmer, completed the task 55.8% faster than the control group.
Source-stated
The authors say observed differences in effects suggest AI pair programmers may help people transition into software development careers; the abstract does not report career outcomes.
Observed heterogenous effects show promise for AI pair programmers to help people transition into software development careers.
Source and contributor context
Peng, Sida, Kalliamvakou, Eirini, Cihon, Peter, Demirer, Mert · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 675eaa23-c5ad-4504-8348-11f0b6972a8e.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-20T12:27:28.052Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: f62bd92b1022d62234cca6fa1ea3e91336ff21df667a55db05e40cbb57f06b56. Source text hash: 1bb5aefbaf44086e3ecaded2d45219cb9dd29700dd005839a2bf9abf6129cdbe.
Where this evidence matters
- Does AI reduce the effort needed to finish work well? · direct · supports
The arrival is the arXiv abstract of Peng, Kalliamvakou, Cihon and Demirer's controlled GitHub Copilot experiment, submitted 13 February 2023. It is a new evidence family, not a report of any foundation or adopted study. It has the same population, task and outcome type as the guide: developers, coding, and comparative time to complete work. What the source observes: recruited developers implemented one specified HTTP server in JavaScript, and the group with AI access completed it 55.8% faster than the control group. This is a comparative measurement, not a self-report. What it bears on: the main question's claim that AI saves effort where the task is defined and checkable. It also bears on quality-useful-work-expands and on software-whole-tasks, which concerns delegating complete, well-specified changes. Together with METR's early-2025 slowdown in familiar, mature repositories, it fits the guide's scope distinction. Gains appear on a bounded new task and can reverse on complex familiar work. Limits: the abstract measures speed on a single task with early-2023 tools. It does not state the quality standard or acceptance check, and it does not count later review, integration or repair. So it does not meet the quality-useful-work-expands signpost of lower total effort at stable quality. Nor does it meet the revision trigger asking for defects after handoff. The authors' remark about career transitions is their interpretation of heterogeneous effects. The abstract reports no career outcomes, so we do not link it to entry or progression. The strongest rival is the guide's first alternative: favorable task selection explains apparent gains. A single greenfield task chosen for the experiment is exactly the kind of setting that rival predicts will flatter AI.
Read the revision from 2026-11-20 - Checkable digital work takes less effort to finish · direct · supports
The arrival is the arXiv abstract of Peng, Kalliamvakou, Cihon and Demirer's controlled GitHub Copilot experiment, submitted 13 February 2023. It is a new evidence family, not a report of any foundation or adopted study. It has the same population, task and outcome type as the guide: developers, coding, and comparative time to complete work. What the source observes: recruited developers implemented one specified HTTP server in JavaScript, and the group with AI access completed it 55.8% faster than the control group. This is a comparative measurement, not a self-report. What it bears on: the main question's claim that AI saves effort where the task is defined and checkable. It also bears on quality-useful-work-expands and on software-whole-tasks, which concerns delegating complete, well-specified changes. Together with METR's early-2025 slowdown in familiar, mature repositories, it fits the guide's scope distinction. Gains appear on a bounded new task and can reverse on complex familiar work. Limits: the abstract measures speed on a single task with early-2023 tools. It does not state the quality standard or acceptance check, and it does not count later review, integration or repair. So it does not meet the quality-useful-work-expands signpost of lower total effort at stable quality. Nor does it meet the revision trigger asking for defects after handoff. The authors' remark about career transitions is their interpretation of heterogeneous effects. The abstract reports no career outcomes, so we do not link it to entry or progression. The strongest rival is the guide's first alternative: favorable task selection explains apparent gains. A single greenfield task chosen for the experiment is exactly the kind of setting that rival predicts will flatter AI.
Read the revision from 2026-11-20 - More whole tasks, with review at the finish · direct · supports
The arrival is the arXiv abstract of Peng, Kalliamvakou, Cihon and Demirer's controlled GitHub Copilot experiment, submitted 13 February 2023. It is a new evidence family, not a report of any foundation or adopted study. It has the same population, task and outcome type as the guide: developers, coding, and comparative time to complete work. What the source observes: recruited developers implemented one specified HTTP server in JavaScript, and the group with AI access completed it 55.8% faster than the control group. This is a comparative measurement, not a self-report. What it bears on: the main question's claim that AI saves effort where the task is defined and checkable. It also bears on quality-useful-work-expands and on software-whole-tasks, which concerns delegating complete, well-specified changes. Together with METR's early-2025 slowdown in familiar, mature repositories, it fits the guide's scope distinction. Gains appear on a bounded new task and can reverse on complex familiar work. Limits: the abstract measures speed on a single task with early-2023 tools. It does not state the quality standard or acceptance check, and it does not count later review, integration or repair. So it does not meet the quality-useful-work-expands signpost of lower total effort at stable quality. Nor does it meet the revision trigger asking for defects after handoff. The authors' remark about career transitions is their interpretation of heterogeneous effects. The abstract reports no career outcomes, so we do not link it to entry or progression. The strongest rival is the guide's first alternative: favorable task selection explains apparent gains. A single greenfield task chosen for the experiment is exactly the kind of setting that rival predicts will flatter AI.
Read the revision from 2026-11-20
Revision and source-check history
Version 1 · 2026-11-20T12:27:28.052Z
arXiv: The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
Authors Sida Peng, Eirini Kalliamvakou, Peter Cihon and Mert Demirer report a controlled experiment testing whether GitHub Copilot, described as an AI pair programmer, helped software developers complete a programming task faster. Recruited developers were asked to implement an HTTP server in JavaScript as quickly as possible. The authors say the group with access to Copilot completed the task 55.8% faster than the control group. This result concerns performance on that task in the experiment; it does not by itself establish how much Copilot changes productivity across other kinds of software development. The authors also report heterogenous effects and say these show promise for helping people transition into software development careers. The abstract does not establish whether that promise translates into better career outcomes.
Original source
· Article
Source details
Original source date: Feb 13, 2023 (2023-02-13).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-20T12:27:28.052Z. First observed: 2026-11-20T03:27:16.171Z. Retrieved: 2026-11-20T12:26:50.782Z.
Limits
The reported task was implementing an HTTP server in JavaScript; the abstract does not establish that the result applies to other software development tasks. The abstract does not give details of how the observed heterogeneous effects varied or report outcomes for people transitioning into software development careers.
Open question
How do the observed effects vary across developers, and do they extend to software development tasks beyond implementing an HTTP server in JavaScript?
Source-stated
In the controlled experiment described by the authors, developers with access to GitHub Copilot completed the assigned task faster than the control group.
The treatment group, with access to the AI pair programmer, completed the task 55.8% faster than the control group.
Source-stated
The authors say observed differences in effects suggest AI pair programmers may help people transition into software development careers; the abstract does not report career outcomes.
Observed heterogenous effects show promise for AI pair programmers to help people transition into software development careers.
Source and contributor context
Peng, Sida, Kalliamvakou, Eirini, Cihon, Peter, Demirer, Mert · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 675eaa23-c5ad-4504-8348-11f0b6972a8e.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-20T12:27:28.052Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: f62bd92b1022d62234cca6fa1ea3e91336ff21df667a55db05e40cbb57f06b56. Source text hash: 1bb5aefbaf44086e3ecaded2d45219cb9dd29700dd005839a2bf9abf6129cdbe.
passed · 2026-11-20T12:27:28.052Z · anthropic/claude-opus-5.5
Publication: 675eaa23-c5ad-4504-8348-11f0b6972a8e
attribution — pass: The reading correctly names the four authors listed on the arXiv page and keeps the 55.8% result and the career-transition claim attributed to them. It does not present either as established fact or as Escape Meridian's view.
Authors:Sida Peng, Eirini Kalliamvakou, Peter Cihon, Mert Demirer
summary — pass: The summary leads with the actor and the experiment, explains Copilot as an AI pair programmer, and states the task and result. It separates the authors' stated promise from demonstrated career outcomes without weakening their claim. The question raises a real open issue that follows from the evidence. 'Heterogenous effects' appears in the summary without a plain gloss, but the claims gloss it as differences in effects, and the term can carry a tooltip, so this does not block understanding.
This paper presents results from a controlled experiment with GitHub Copilot, an AI pair programmer.
date — pass: originalPublished is 2023-02-13. This matches the arXiv v1 submission date, and only one version is listed.
[Submitted on 13 Feb 2023]
support — pass: Both claims match the abstract word for word in substance. The limitations correctly note that the result covers one task, and that the abstract reports no detail on how effects varied and no career outcomes. No benefits, costs or stakeholders are invented, and the test result is not turned into general real-world productivity.
Recruited software developers were asked to implement an HTTP server in JavaScript as quickly as possible.
media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct way to reach the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv