# arXiv cs.AI: Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3

[Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-can-llms-improve-their-own-harness-a-controlled-eead71e4)

In a paper on arXiv cs.AI, Jens Lehmann and Sahar Vahdati study whether frontier models can build and improve their own harness: a symbolic program that coordinates a model’s perception, memory, tool use, recovery, and inference allocation. In their ARC-AGI-3 experiment, each model built a task harness, tried it on five development games, and revised its codebase twice; the researchers evaluated every version on twenty games that did not guide adaptation. Across 27 lineages, performance ranged from zero to near-ceiling scores, but two revisions raised mean held-out RHAE by only 2.80 points, with a 95% interval from -1.44 to 7.35. The authors say development gains overstated improvement on held-out games, and lower inference cost could come at the expense of task performance. They also report that no revision retained its parent. A post hoc rule for selecting among existing snapshots by development score raised the held-out gain to 4.53 points, which the authors frame as a distinction between generating revisions and deciding which to deploy. They conclude that, under this two-revision protocol, strong task performance does not ensure that later revisions generalize or preserve existing capabilities. The abstract notes that human-steered systems show further headroom; it does not quantify that headroom.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv cs.AI · Article

Original source date: Oct 21, 2026 (2026-10-21).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-16T07:48:46.801Z. First observed: 2026-10-23T06:25:30.514Z. Retrieved: 2026-11-16T07:47:48.555Z.

- [Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3](https://arxiv.org/abs/2610.30499)

## Limits

The results concern ARC-AGI-3 and a protocol in which models revised the codebase twice; the abstract does not establish how results would change under other tasks or revision protocols. The abstract reports that human-steered systems demonstrate further headroom but does not quantify that headroom. The reported figures concern the study’s RHAE measure and inference cost.


## Open question

Would additional revisions improve held-out RHAE while preserving existing capabilities under this protocol?


## Source-stated

The authors define a harness as a symbolic program that coordinates several aspects of a language model’s operation.

> A harness is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model's behalf.

- [Supporting source](https://arxiv.org/abs/2610.30499)

## Source-stated

The experiment had each model build a harness, act on five development games, and revise the codebase twice; every version was evaluated on twenty games that did not influence adaptation.

> acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation.

- [Supporting source](https://arxiv.org/abs/2610.30499)

## Source-stated

Across 27 lineages, two revisions raised mean held-out RHAE by 2.80 points, with a 95% interval from -1.44 to 7.35.

> Across 27 lineages, harness performance ranges from zero to near-ceiling scores. Two revisions raise mean held-out RHAE by only 2.80 points (95% interval \[-1.44,7.35\]).

- [Supporting source](https://arxiv.org/abs/2610.30499)

## Source-stated

The authors report that development gains overstated held-out improvement and that lower inference cost could come at the expense of task performance.

> Development gains overstate held-out improvement, and reductions in inference cost can come at the expense of task performance.

- [Supporting source](https://arxiv.org/abs/2610.30499)

## Source-stated

No revision retained its parent, while a post hoc selection rule based on development score raised the held-out gain to 4.53 points.

> No revision retains its parent, yet a post hoc rule that selects among existing snapshots by development score raises the held-out gain to 4.53 points (\[0.81,8.68\])

- [Supporting source](https://arxiv.org/abs/2610.30499)

## Source-stated

The authors identify a gap between autonomous harness construction and reliable adaptation under the specific two-revision protocol studied.

> These findings reveal a gap between autonomous harness construction and reliable adaptation under this two-revision protocol

- [Supporting source](https://arxiv.org/abs/2610.30499)

## Source and contributor context

Lehmann, Jens, Vahdati, Sahar · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## Frontier model

One of the most capable AI models available at a given time; the set shifts as models improve.

- [Definition reference](https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper)

## Tool use

A model's ability to operate other software, such as a search engine, a calculator or a code runner, while it works.

- [Definition reference](https://docs.anthropic.com/en/docs/build-with-claude/tool-use/overview)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: fdcfb83b-921b-4a5b-8078-803fb93907b9.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-16T07:48:46.801Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 198a8a2c3f86a3cadcf04aaf7db1b501ad09863ba073dab0736a7dd50d428c79. Source text hash: d9fd0c4cb9d3db238cdc34eff875ab5cbb6ccf73a5257c2543ff10885f0e39d8.


## Revision and source-check history

### Version 1 — 2026-11-16T07:48:46.801Z

In a paper on arXiv cs.AI, Jens Lehmann and Sahar Vahdati study whether frontier models can build and improve their own harness: a symbolic program that coordinates a model’s perception, memory, tool use, recovery, and inference allocation. In their ARC-AGI-3 experiment, each model built a task harness, tried it on five development games, and revised its codebase twice; the researchers evaluated every version on twenty games that did not guide adaptation. Across 27 lineages, performance ranged from zero to near-ceiling scores, but two revisions raised mean held-out RHAE by only 2.80 points, with a 95% interval from -1.44 to 7.35. The authors say development gains overstated improvement on held-out games, and lower inference cost could come at the expense of task performance. They also report that no revision retained its parent. A post hoc rule for selecting among existing snapshots by development score raised the held-out gain to 4.53 points, which the authors frame as a distinction between generating revisions and deciding which to deploy. They conclude that, under this two-revision protocol, strong task performance does not ensure that later revisions generalize or preserve existing capabilities. The abstract notes that human-steered systems show further headroom; it does not quantify that headroom.

## Original source

arXiv cs.AI · Article

Original source date: Oct 21, 2026 (2026-10-21).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-16T07:48:46.801Z. First observed: 2026-10-23T06:25:30.514Z. Retrieved: 2026-11-16T07:47:48.555Z.

- [Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3](https://arxiv.org/abs/2610.30499)

## Limits

The results concern ARC-AGI-3 and a protocol in which models revised the codebase twice; the abstract does not establish how results would change under other tasks or revision protocols. The abstract reports that human-steered systems demonstrate further headroom but does not quantify that headroom. The reported figures concern the study’s RHAE measure and inference cost.


## Open question

Would additional revisions improve held-out RHAE while preserving existing capabilities under this protocol?


## Source-stated

The authors define a harness as a symbolic program that coordinates several aspects of a language model’s operation.

> A harness is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model's behalf.

- [Supporting source](https://arxiv.org/abs/2610.30499)

## Source-stated

The experiment had each model build a harness, act on five development games, and revise the codebase twice; every version was evaluated on twenty games that did not influence adaptation.

> acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation.

- [Supporting source](https://arxiv.org/abs/2610.30499)

## Source-stated

Across 27 lineages, two revisions raised mean held-out RHAE by 2.80 points, with a 95% interval from -1.44 to 7.35.

> Across 27 lineages, harness performance ranges from zero to near-ceiling scores. Two revisions raise mean held-out RHAE by only 2.80 points (95% interval \[-1.44,7.35\]).

- [Supporting source](https://arxiv.org/abs/2610.30499)

## Source-stated

The authors report that development gains overstated held-out improvement and that lower inference cost could come at the expense of task performance.

> Development gains overstate held-out improvement, and reductions in inference cost can come at the expense of task performance.

- [Supporting source](https://arxiv.org/abs/2610.30499)

## Source-stated

No revision retained its parent, while a post hoc selection rule based on development score raised the held-out gain to 4.53 points.

> No revision retains its parent, yet a post hoc rule that selects among existing snapshots by development score raises the held-out gain to 4.53 points (\[0.81,8.68\])

- [Supporting source](https://arxiv.org/abs/2610.30499)

## Source-stated

The authors identify a gap between autonomous harness construction and reliable adaptation under the specific two-revision protocol studied.

> These findings reveal a gap between autonomous harness construction and reliable adaptation under this two-revision protocol

- [Supporting source](https://arxiv.org/abs/2610.30499)

## Source and contributor context

Lehmann, Jens, Vahdati, Sahar · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## Frontier model

One of the most capable AI models available at a given time; the set shifts as models improve.

- [Definition reference](https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper)

## Tool use

A model's ability to operate other software, such as a search engine, a calculator or a code runner, while it works.

- [Definition reference](https://docs.anthropic.com/en/docs/build-with-claude/tool-use/overview)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: fdcfb83b-921b-4a5b-8078-803fb93907b9.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-16T07:48:46.801Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 198a8a2c3f86a3cadcf04aaf7db1b501ad09863ba073dab0736a7dd50d428c79. Source text hash: d9fd0c4cb9d3db238cdc34eff875ab5cbb6ccf73a5257c2543ff10885f0e39d8.


### uncertain — 2026-11-16T07:48:46.801Z — anthropic/claude-opus-5.5

Publication: fdcfb83b-921b-4a5b-8078-803fb93907b9

attribution — pass: The reading attributes the study to Jens Lehmann and Sahar Vahdati in a paper on arXiv cs.AI. This matches the author list and listing. Findings are kept as the authors' reports rather than established facts.

> Authors:Jens Lehmann, Sahar Vahdati

summary — uncertain: The summary is faithful but hard for non-specialists to follow. Three central terms go unexplained: RHAE (the source expands it as Relative Human Action Efficiency), ARC-AGI-3 (an interactive benchmark where agents explore unfamiliar environments and infer goals), and 'no revision retained its parent'. Two pieces of significance are also dropped: the 4.53 gain's interval \[0.81,8.68\], which excludes zero while the 2.80 interval does not, and the context that human-steered systems already reach ceiling scores. Smallest repair: briefly define these terms and add that interval and context.

> ARC-AGI-3 is an interactive benchmark in which agents must explore unfamiliar environments, figure out their goals on the fly, and keep learning as they go.

date — pass: originalPublished is 2026-10-21. This matches the v1 submission date, not the later v2 revision of 13 Nov 2026 or the feed date.

> \[Submitted on 21 Oct 2026 (v1), last revised 13 Nov 2026 (this version, v2)\]

support — pass: Each claim maps to verbatim abstract text: the harness definition, the protocol, the 2.80-point gain and its interval, development-versus-held-out overstatement, the cost trade-off, the 4.53-point post hoc gain, and the gap under the two-revision protocol. The limitations and the question stay conditional and introduce no unsupported benefits or costs.

> Two revisions raise mean held-out RHAE by only 2.80 points (95% interval \[-1.44,7.35\]).

media — pass: The selected media is the verified direct link to the arXiv abstract page. That page is the original source.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/eead71e400420a50f101f88d99a6b4bf5fcfcdadda4e572741bb9b84916f84f0/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-can-llms-improve-their-own-harness-a-controlled-eead71e4.json)
