Lehmann, Jens, Vahdati, Sahar · Oct 21, 2026
arXiv cs.AI: Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3
Markdown · JSON · Original source
In a paper on cs.AI, Jens Lehmann and Sahar Vahdati study whether can build and improve their own harness: a symbolic program that coordinates a model’s perception, memory, , recovery, and inference allocation. In their ARC-AGI-3 experiment, each model built a task harness, tried it on five development games, and revised its codebase twice; the researchers evaluated every version on twenty games that did not guide adaptation. Across 27 lineages, performance ranged from zero to near-ceiling scores, but two revisions raised mean held-out RHAE by only 2.80 points, with a 95% interval from -1.44 to 7.35. The authors say development gains overstated improvement on held-out games, and lower inference cost could come at the expense of task performance. They also report that no revision retained its parent. A post hoc rule for selecting among existing snapshots by development score raised the held-out gain to 4.53 points, which the authors frame as a distinction between generating revisions and deciding which to deploy. They conclude that, under this two-revision protocol, strong task performance does not ensure that later revisions generalize or preserve existing capabilities. The abstract notes that human-steered systems show further headroom; it does not quantify that headroom.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 21, 2026 (2026-10-21).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-16T07:48:46.801Z. First observed: 2026-10-23T06:25:30.514Z. Retrieved: 2026-11-16T07:47:48.555Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The results concern ARC-AGI-3 and a protocol in which models revised the codebase twice; the abstract does not establish how results would change under other tasks or revision protocols. The abstract reports that human-steered systems demonstrate further headroom but does not quantify that headroom. The reported figures concern the study’s RHAE measure and inference cost.
Open question
Would additional revisions improve held-out RHAE while preserving existing capabilities under this protocol?
Source-stated
The authors define a harness as a symbolic program that coordinates several aspects of a language model’s operation.
A harness is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model's behalf.
Source-stated
The experiment had each model build a harness, act on five development games, and revise the codebase twice; every version was evaluated on twenty games that did not influence adaptation.
acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation.
Source-stated
Across 27 lineages, two revisions raised mean held-out RHAE by 2.80 points, with a 95% interval from -1.44 to 7.35.
Across 27 lineages, harness performance ranges from zero to near-ceiling scores. Two revisions raise mean held-out RHAE by only 2.80 points (95% interval [-1.44,7.35]).
Source-stated
The authors report that development gains overstated held-out improvement and that lower inference cost could come at the expense of task performance.
Development gains overstate held-out improvement, and reductions in inference cost can come at the expense of task performance.
Source-stated
No revision retained its parent, while a post hoc selection rule based on development score raised the held-out gain to 4.53 points.
No revision retains its parent, yet a post hoc rule that selects among existing snapshots by development score raises the held-out gain to 4.53 points ([0.81,8.68])
Source-stated
The authors identify a gap between autonomous harness construction and reliable adaptation under the specific two-revision protocol studied.
These findings reveal a gap between autonomous harness construction and reliable adaptation under this two-revision protocol
Source and contributor context
Lehmann, Jens, Vahdati, Sahar · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
Tool use
A model's ability to operate other software, such as a search engine, a calculator or a code runner, while it works.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: fdcfb83b-921b-4a5b-8078-803fb93907b9.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-16T07:48:46.801Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 198a8a2c3f86a3cadcf04aaf7db1b501ad09863ba073dab0736a7dd50d428c79. Source text hash: d9fd0c4cb9d3db238cdc34eff875ab5cbb6ccf73a5257c2543ff10885f0e39d8.
Revision and source-check history
Version 1 · 2026-11-16T07:48:46.801Z
arXiv cs.AI: Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3
In a paper on cs.AI, Jens Lehmann and Sahar Vahdati study whether can build and improve their own harness: a symbolic program that coordinates a model’s perception, memory, , recovery, and inference allocation. In their ARC-AGI-3 experiment, each model built a task harness, tried it on five development games, and revised its codebase twice; the researchers evaluated every version on twenty games that did not guide adaptation. Across 27 lineages, performance ranged from zero to near-ceiling scores, but two revisions raised mean held-out RHAE by only 2.80 points, with a 95% interval from -1.44 to 7.35. The authors say development gains overstated improvement on held-out games, and lower inference cost could come at the expense of task performance. They also report that no revision retained its parent. A post hoc rule for selecting among existing snapshots by development score raised the held-out gain to 4.53 points, which the authors frame as a distinction between generating revisions and deciding which to deploy. They conclude that, under this two-revision protocol, strong task performance does not ensure that later revisions generalize or preserve existing capabilities. The abstract notes that human-steered systems show further headroom; it does not quantify that headroom.
Original source
cs.AI · Article
Source details
Original source date: Oct 21, 2026 (2026-10-21).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-16T07:48:46.801Z. First observed: 2026-10-23T06:25:30.514Z. Retrieved: 2026-11-16T07:47:48.555Z.
Limits
The results concern ARC-AGI-3 and a protocol in which models revised the codebase twice; the abstract does not establish how results would change under other tasks or revision protocols. The abstract reports that human-steered systems demonstrate further headroom but does not quantify that headroom. The reported figures concern the study’s RHAE measure and inference cost.
Open question
Would additional revisions improve held-out RHAE while preserving existing capabilities under this protocol?
Source-stated
The authors define a harness as a symbolic program that coordinates several aspects of a language model’s operation.
A harness is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model's behalf.
Source-stated
The experiment had each model build a harness, act on five development games, and revise the codebase twice; every version was evaluated on twenty games that did not influence adaptation.
acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation.
Source-stated
Across 27 lineages, two revisions raised mean held-out RHAE by 2.80 points, with a 95% interval from -1.44 to 7.35.
Across 27 lineages, harness performance ranges from zero to near-ceiling scores. Two revisions raise mean held-out RHAE by only 2.80 points (95% interval [-1.44,7.35]).
Source-stated
The authors report that development gains overstated held-out improvement and that lower inference cost could come at the expense of task performance.
Development gains overstate held-out improvement, and reductions in inference cost can come at the expense of task performance.
Source-stated
No revision retained its parent, while a post hoc selection rule based on development score raised the held-out gain to 4.53 points.
No revision retains its parent, yet a post hoc rule that selects among existing snapshots by development score raises the held-out gain to 4.53 points ([0.81,8.68])
Source-stated
The authors identify a gap between autonomous harness construction and reliable adaptation under the specific two-revision protocol studied.
These findings reveal a gap between autonomous harness construction and reliable adaptation under this two-revision protocol
Source and contributor context
Lehmann, Jens, Vahdati, Sahar · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
Tool use
A model's ability to operate other software, such as a search engine, a calculator or a code runner, while it works.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: fdcfb83b-921b-4a5b-8078-803fb93907b9.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-16T07:48:46.801Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 198a8a2c3f86a3cadcf04aaf7db1b501ad09863ba073dab0736a7dd50d428c79. Source text hash: d9fd0c4cb9d3db238cdc34eff875ab5cbb6ccf73a5257c2543ff10885f0e39d8.
uncertain · 2026-11-16T07:48:46.801Z · anthropic/claude-opus-5.5
Publication: fdcfb83b-921b-4a5b-8078-803fb93907b9
attribution — pass: The reading attributes the study to Jens Lehmann and Sahar Vahdati in a paper on arXiv cs.AI. This matches the author list and listing. Findings are kept as the authors' reports rather than established facts.
Authors:Jens Lehmann, Sahar Vahdati
summary — uncertain: The summary is faithful but hard for non-specialists to follow. Three central terms go unexplained: RHAE (the source expands it as Relative Human Action Efficiency), ARC-AGI-3 (an interactive benchmark where agents explore unfamiliar environments and infer goals), and 'no revision retained its parent'. Two pieces of significance are also dropped: the 4.53 gain's interval [0.81,8.68], which excludes zero while the 2.80 interval does not, and the context that human-steered systems already reach ceiling scores. Smallest repair: briefly define these terms and add that interval and context.
ARC-AGI-3 is an interactive benchmark in which agents must explore unfamiliar environments, figure out their goals on the fly, and keep learning as they go.
date — pass: originalPublished is 2026-10-21. This matches the v1 submission date, not the later v2 revision of 13 Nov 2026 or the feed date.
[Submitted on 21 Oct 2026 (v1), last revised 13 Nov 2026 (this version, v2)]
support — pass: Each claim maps to verbatim abstract text: the harness definition, the protocol, the 2.80-point gain and its interval, development-versus-held-out overstatement, the cost trade-off, the 4.53-point post hoc gain, and the gap under the two-revision protocol. The limitations and the question stay conditional and introduce no unsupported benefits or costs.
Two revisions raise mean held-out RHAE by only 2.80 points (95% interval [-1.44,7.35]).
media — pass: The selected media is the verified direct link to the arXiv abstract page. That page is the original source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceA model's ability to operate other software, such as a search engine, a calculator or a code runner, while it works.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceA model's ability to operate other software, such as a search engine, a calculator or a code runner, while it works.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv