# OpenAI Codex and Novice Learners in Introductory Programming

[Canonical reading](https://escapemeridian.com/readings/openai-codex-and-novice-learners-in-introductory-programming-d0b2b6b4)

The paper’s authors report that access to OpenAI Codex improved novice learners’ performance on code-authoring tasks in a controlled experiment, without reducing their performance on later code-modification tasks. The result for learning over time was less clear: learners with Codex access did slightly better on evaluation post-tests one week later, but the difference did not reach statistical significance. The authors also report that learners with higher Scratch pre-test scores performed significantly better on retention post-tests if they had prior access to Codex. Together, these findings distinguish performance while authoring code from evidence about what learners retained: the authors report improved task performance, but the overall post-test result does not establish a retention benefit. The Scratch-related result points to a difference among learners in this experiment, though the abstract does not explain why it occurred. The study involved novice learners completing Python code-authoring and code-modification tasks; whether these results apply beyond those learners and tasks remains unknown.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv · Article

Original source date: Feb 15, 2023 (2023-02-15).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-20T14:08:58.244Z. First observed: 2026-11-20T13:50:23.739Z. Retrieved: 2026-11-20T13:53:36.030Z.

- [Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming](https://arxiv.org/abs/2302.07427)

## Limits

The reported experiment involved 69 novices ages 10–17 completing Python tasks; the abstract does not establish whether the results generalize to other learners or settings. The overall post-test difference one week later did not reach statistical significance, so the abstract does not establish an overall retention benefit. The abstract reports the Scratch-related retention result but does not explain what accounts for it.


## Open question

What accounts for the higher retention post-test performance among learners with higher Scratch pre-test scores who had prior access to Codex?


## Source-stated

The authors report that access to Codex improved code-authoring performance and did not reduce performance on manual code-modification tasks in their experiment.

> using Codex significantly increased code-authoring performance (1.15x increased completion rate and 1.8x higher scores) while not decreasing performance on manual code-modification tasks

- [Supporting source](https://arxiv.org/abs/2302.07427)

## Source-stated

Learners with Codex access did slightly better on post-tests one week later, but the difference was not statistically significant.

> learners with access to Codex during the training phase performed slightly better on the evaluation post-tests conducted one week later, although this difference did not reach statistical significance

- [Supporting source](https://arxiv.org/abs/2302.07427)

## Source-stated

The authors report a significant retention post-test advantage associated with prior Codex access among learners with higher Scratch pre-test scores.

> learners with higher Scratch pre-test scores performed significantly better on retention post-tests, if they had prior access to Codex.

- [Supporting source](https://arxiv.org/abs/2302.07427)

## Source and contributor context

Kazemitabaar, Majeed, Chow, Justin, Ma, Carl Ka To, Ericson, Barbara J., Weintrop, David, Grossman, Tovi · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Statistical significance

A test of whether a result is unlikely to be due to chance alone; it does not say how large or important an effect is.

- [Definition reference](https://en.wikipedia.org/wiki/Statistical_significance)

## OpenAI

OpenAI develops AI models and products and publishes research, release announcements and evaluations.

- [Explore this context](/entities/openai)
- [Reference](https://openai.com/news/)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## TIME

TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.

- [Explore this context](/entities/time)
- [Reference](https://time.com/)

## Editorial provenance

Version 1. Publication ID: ad99cda8-12a2-44bf-896b-690f955e239f.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-20T14:08:58.244Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: fcc9b0f7223add7d23262d7e5b0770b37ea065125ddd64e6b4a4b50536fffe8a. Source text hash: 8b4cd1efc2ef5e072fd69c1416f2baf05af43882bc45374258fd29c58645986f.


## Revision and source-check history

### Version 1 — 2026-11-20T14:08:58.244Z

The paper’s authors report that access to OpenAI Codex improved novice learners’ performance on code-authoring tasks in a controlled experiment, without reducing their performance on later code-modification tasks. The result for learning over time was less clear: learners with Codex access did slightly better on evaluation post-tests one week later, but the difference did not reach statistical significance. The authors also report that learners with higher Scratch pre-test scores performed significantly better on retention post-tests if they had prior access to Codex. Together, these findings distinguish performance while authoring code from evidence about what learners retained: the authors report improved task performance, but the overall post-test result does not establish a retention benefit. The Scratch-related result points to a difference among learners in this experiment, though the abstract does not explain why it occurred. The study involved novice learners completing Python code-authoring and code-modification tasks; whether these results apply beyond those learners and tasks remains unknown.

## Original source

arXiv · Article

Original source date: Feb 15, 2023 (2023-02-15).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-20T14:08:58.244Z. First observed: 2026-11-20T13:50:23.739Z. Retrieved: 2026-11-20T13:53:36.030Z.

- [Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming](https://arxiv.org/abs/2302.07427)

## Limits

The reported experiment involved 69 novices ages 10–17 completing Python tasks; the abstract does not establish whether the results generalize to other learners or settings. The overall post-test difference one week later did not reach statistical significance, so the abstract does not establish an overall retention benefit. The abstract reports the Scratch-related retention result but does not explain what accounts for it.


## Open question

What accounts for the higher retention post-test performance among learners with higher Scratch pre-test scores who had prior access to Codex?


## Source-stated

The authors report that access to Codex improved code-authoring performance and did not reduce performance on manual code-modification tasks in their experiment.

> using Codex significantly increased code-authoring performance (1.15x increased completion rate and 1.8x higher scores) while not decreasing performance on manual code-modification tasks

- [Supporting source](https://arxiv.org/abs/2302.07427)

## Source-stated

Learners with Codex access did slightly better on post-tests one week later, but the difference was not statistically significant.

> learners with access to Codex during the training phase performed slightly better on the evaluation post-tests conducted one week later, although this difference did not reach statistical significance

- [Supporting source](https://arxiv.org/abs/2302.07427)

## Source-stated

The authors report a significant retention post-test advantage associated with prior Codex access among learners with higher Scratch pre-test scores.

> learners with higher Scratch pre-test scores performed significantly better on retention post-tests, if they had prior access to Codex.

- [Supporting source](https://arxiv.org/abs/2302.07427)

## Source and contributor context

Kazemitabaar, Majeed, Chow, Justin, Ma, Carl Ka To, Ericson, Barbara J., Weintrop, David, Grossman, Tovi · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Statistical significance

A test of whether a result is unlikely to be due to chance alone; it does not say how large or important an effect is.

- [Definition reference](https://en.wikipedia.org/wiki/Statistical_significance)

## OpenAI

OpenAI develops AI models and products and publishes research, release announcements and evaluations.

- [Explore this context](/entities/openai)
- [Reference](https://openai.com/news/)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## TIME

TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.

- [Explore this context](/entities/time)
- [Reference](https://time.com/)

## Editorial provenance

Version 1. Publication ID: ad99cda8-12a2-44bf-896b-690f955e239f.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-20T14:08:58.244Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: fcc9b0f7223add7d23262d7e5b0770b37ea065125ddd64e6b4a4b50536fffe8a. Source text hash: 8b4cd1efc2ef5e072fd69c1416f2baf05af43882bc45374258fd29c58645986f.


### uncertain — 2026-11-20T14:08:58.244Z — anthropic/claude-opus-5.5

Publication: ad99cda8-12a2-44bf-896b-690f955e239f

attribution — pass: The reading attributes the findings to the paper's authors, who match the arXiv listing. It keeps them as reported results rather than established fact.

> Authors:Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J. Ericson, David Weintrop, Tovi Grossman

summary — uncertain: The reading is faithful but not self-contained. It never says what OpenAI Codex is: an AI code generator that writes code from plain-language descriptions. It also never explains Scratch. It omits the over-reliance concern behind the study. The learners' ages (10-17) and the study size appear only in the limitations. Smallest repair: add one clause saying Codex generates code from natural-language descriptions, and one clause stating the over-reliance concern the study tested. Separately, consider a term-library tooltip for Scratch.

> AI code generators like OpenAI Codex have the potential to assist novice programmers by generating code from natural language descriptions, however, over-reliance might negatively impact learning and retention.

date — pass: originalPublished is 2023-02-15, which matches the v1 submission date rather than the later v2 revision.

> Submitted on 15 Feb 2023 (v1), last revised 21 Feb 2023 (this version, v2)

support — pass: Each claim matches the abstract. The non-significant retention result is kept as non-significant, and the Scratch subgroup result is not overexplained. The limitations and the question follow from the evidence without inventing benefits or causes.

> although this difference did not reach statistical significance

media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the original source.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/d0b2b6b4594ae2bcebfb065f6881e9a7acffee7e73660aafb911135c166ee0a8/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/openai-codex-and-novice-learners-in-introductory-programming-d0b2b6b4.json)
