{"id":"eead71e400420a50f101f88d99a6b4bf5fcfcdadda4e572741bb9b84916f84f0","title":"arXiv cs.AI: Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3","argument":"In a paper on arXiv cs.AI, Jens Lehmann and Sahar Vahdati study whether frontier models can build and improve their own harness: a symbolic program that coordinates a model’s perception, memory, tool use, recovery, and inference allocation. In their ARC-AGI-3 experiment, each model built a task harness, tried it on five development games, and revised its codebase twice; the researchers evaluated every version on twenty games that did not guide adaptation. Across 27 lineages, performance ranged from zero to near-ceiling scores, but two revisions raised mean held-out RHAE by only 2.80 points, with a 95% interval from -1.44 to 7.35. The authors say development gains overstated improvement on held-out games, and lower inference cost could come at the expense of task performance. They also report that no revision retained its parent. A post hoc rule for selecting among existing snapshots by development score raised the held-out gain to 4.53 points, which the authors frame as a distinction between generating revisions and deciding which to deploy. They conclude that, under this two-revision protocol, strong task performance does not ensure that later revisions generalize or preserve existing capabilities. The abstract notes that human-steered systems show further headroom; it does not quantify that headroom.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The results concern ARC-AGI-3 and a protocol in which models revised the codebase twice; the abstract does not establish how results would change under other tasks or revision protocols. The abstract reports that human-steered systems demonstrate further headroom but does not quantify that headroom. The reported figures concern the study’s RHAE measure and inference cost.","speaker":{"name":"Lehmann, Jens, Vahdati, Sahar","roleAtSource":"Article author"},"source":{"title":"Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.30499","publishedAt":"2026-10-21","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors define a harness as a symbolic program that coordinates several aspects of a language model’s operation.","excerpt":"A harness is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model's behalf.","url":"https://arxiv.org/abs/2610.30499","relationship":"Source-stated"},{"text":"The experiment had each model build a harness, act on five development games, and revise the codebase twice; every version was evaluated on twenty games that did not influence adaptation.","excerpt":"acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation.","url":"https://arxiv.org/abs/2610.30499","relationship":"Source-stated"},{"text":"Across 27 lineages, two revisions raised mean held-out RHAE by 2.80 points, with a 95% interval from -1.44 to 7.35.","excerpt":"Across 27 lineages, harness performance ranges from zero to near-ceiling scores. Two revisions raise mean held-out RHAE by only 2.80 points (95% interval [-1.44,7.35]).","url":"https://arxiv.org/abs/2610.30499","relationship":"Source-stated"},{"text":"The authors report that development gains overstated held-out improvement and that lower inference cost could come at the expense of task performance.","excerpt":"Development gains overstate held-out improvement, and reductions in inference cost can come at the expense of task performance.","url":"https://arxiv.org/abs/2610.30499","relationship":"Source-stated"},{"text":"No revision retained its parent, while a post hoc selection rule based on development score raised the held-out gain to 4.53 points.","excerpt":"No revision retains its parent, yet a post hoc rule that selects among existing snapshots by development score raises the held-out gain to 4.53 points ([0.81,8.68])","url":"https://arxiv.org/abs/2610.30499","relationship":"Source-stated"},{"text":"The authors identify a gap between autonomous harness construction and reliable adaptation under the specific two-revision protocol studied.","excerpt":"These findings reveal a gap between autonomous harness construction and reliable adaptation under this two-revision protocol","url":"https://arxiv.org/abs/2610.30499","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.30499","checkedAt":"2026-11-16T08:52:06.834Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Would additional revisions improve held-out RHAE while preserving existing capabilities under this protocol?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3","summary":"In a paper on arXiv cs.AI, Jens Lehmann and Sahar Vahdati study whether frontier models can build and improve their own harness: a symbolic program that coordinates a model’s perception, memory, tool use, recovery, and inference allocation. In their ARC-AGI-3 experiment, each model built a task harness, tried it on five development games, and revised its codebase twice; the researchers evaluated every version on twenty games that did not guide adaptation. Across 27 lineages, performance ranged from zero to near-ceiling scores, but two revisions raised mean held-out RHAE by only 2.80 points, with a 95% interval from -1.44 to 7.35. The authors say development gains overstated improvement on held-out games, and lower inference cost could come at the expense of task performance. They also report that no revision retained its parent. A post hoc rule for selecting among existing snapshots by development score raised the held-out gain to 4.53 points, which the authors frame as a distinction between generating revisions and deciding which to deploy. They conclude that, under this two-revision protocol, strong task performance does not ensure that later revisions generalize or preserve existing capabilities. The abstract notes that human-steered systems show further headroom; it does not quantify that headroom.","question":"Would additional revisions improve held-out RHAE while preserving existing capabilities under this protocol?","claims":[{"text":"The authors define a harness as a symbolic program that coordinates several aspects of a language model’s operation.","excerpt":"A harness is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model's behalf.","support":"source-stated"},{"text":"The experiment had each model build a harness, act on five development games, and revise the codebase twice; every version was evaluated on twenty games that did not influence adaptation.","excerpt":"acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation.","support":"source-stated"},{"text":"Across 27 lineages, two revisions raised mean held-out RHAE by 2.80 points, with a 95% interval from -1.44 to 7.35.","excerpt":"Across 27 lineages, harness performance ranges from zero to near-ceiling scores. Two revisions raise mean held-out RHAE by only 2.80 points (95% interval [-1.44,7.35]).","support":"source-stated"},{"text":"The authors report that development gains overstated held-out improvement and that lower inference cost could come at the expense of task performance.","excerpt":"Development gains overstate held-out improvement, and reductions in inference cost can come at the expense of task performance.","support":"source-stated"},{"text":"No revision retained its parent, while a post hoc selection rule based on development score raised the held-out gain to 4.53 points.","excerpt":"No revision retains its parent, yet a post hoc rule that selects among existing snapshots by development score raises the held-out gain to 4.53 points ([0.81,8.68])","support":"source-stated"},{"text":"The authors identify a gap between autonomous harness construction and reliable adaptation under the specific two-revision protocol studied.","excerpt":"These findings reveal a gap between autonomous harness construction and reliable adaptation under this two-revision protocol","support":"source-stated"}],"limitations":["The results concern ARC-AGI-3 and a protocol in which models revised the codebase twice; the abstract does not establish how results would change under other tasks or revision protocols.","The abstract reports that human-steered systems demonstrate further headroom but does not quantify that headroom. The reported figures concern the study’s RHAE measure and inference cost."],"mediaId":"f00330389119ea4b3639d2d143b9de933a62cf78f781fc89a17fec8a1a785a2c","correctionReason":null,"id":"eead71e400420a50f101f88d99a6b4bf5fcfcdadda4e572741bb9b84916f84f0","publicationId":"fdcfb83b-921b-4a5b-8078-803fb93907b9","version":1,"sourceUrl":"https://arxiv.org/abs/2610.30499","sourceTitle":"Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-21","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-23T06:25:30.514Z","createdAt":"2026-11-16T07:48:46.801Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"f00330389119ea4b3639d2d143b9de933a62cf78f781fc89a17fec8a1a785a2c","kind":"article","url":"https://arxiv.org/abs/2610.30499","sourceUrl":"https://arxiv.org/abs/2610.30499","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-16T07:47:48.865Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.30499","checkedAt":"2026-11-16T08:52:06.834Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"f00330389119ea4b3639d2d143b9de933a62cf78f781fc89a17fec8a1a785a2c","confidence":0.96,"provider":"typesafe","model":"jev-1.13.0","inputHash":"79aeed8ba40a48b9dbc481f1eace310305dc92a9d89ae20cd2404220cbaccfef","decidedAt":"2026-11-16T07:47:53.716Z"},"provenance":{"textHash":"d9fd0c4cb9d3db238cdc34eff875ab5cbb6ccf73a5257c2543ff10885f0e39d8","observationId":"198a8a2c3f86a3cadcf04aaf7db1b501ad09863ba073dab0736a7dd50d428c79","retrievedAt":"2026-11-16T07:47:48.555Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"78eac4f8-23a9-4578-8cea-55d206496e2a"},"reviewer":{"model":"gpt-6-luna","runId":"68d57e78-7cf6-4535-8e20-a1172f2f03ee"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-16T07:48:46.801Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Lehmann, Jens","url":null,"kind":"unknown"},{"name":"Vahdati, Sahar","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Lehmann, Jens","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Lehmann, Jens","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.30499","locator":"head/meta[name=author|citation_author]"},"assertionId":"b29130cac6e029e0d1929384195cab8813cbfaa8cc6b4944ff0a28b24b4c58cf:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-16T08:52:06.834Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b29130cac6e029e0d1929384195cab8813cbfaa8cc6b4944ff0a28b24b4c58cf","sourceUrl":"https://arxiv.org/abs/2610.30499","contentSha256":"471ab0a08a2561486599879c11699b93ea860eb68747fe2ea2199df7e55d90e5","textHash":"d9fd0c4cb9d3db238cdc34eff875ab5cbb6ccf73a5257c2543ff10885f0e39d8","observedAt":"2026-11-16T08:52:06.834Z"}},{"actor":{"name":"Vahdati, Sahar","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Vahdati, Sahar","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.30499","locator":"head/meta[name=author|citation_author]"},"assertionId":"b29130cac6e029e0d1929384195cab8813cbfaa8cc6b4944ff0a28b24b4c58cf:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-16T08:52:06.834Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b29130cac6e029e0d1929384195cab8813cbfaa8cc6b4944ff0a28b24b4c58cf","sourceUrl":"https://arxiv.org/abs/2610.30499","contentSha256":"471ab0a08a2561486599879c11699b93ea860eb68747fe2ea2199df7e55d90e5","textHash":"d9fd0c4cb9d3db238cdc34eff875ab5cbb6ccf73a5257c2543ff10885f0e39d8","observedAt":"2026-11-16T08:52:06.834Z"}}],"participationCoverage":{"total":2,"returned":2,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.30499","snapshotId":"b29130cac6e029e0d1929384195cab8813cbfaa8cc6b4944ff0a28b24b4c58cf","contentSha256":"471ab0a08a2561486599879c11699b93ea860eb68747fe2ea2199df7e55d90e5","textHash":"d9fd0c4cb9d3db238cdc34eff875ab5cbb6ccf73a5257c2543ff10885f0e39d8","observedAt":"2026-11-16T08:52:06.834Z","basis":"publisher-page-metadata","sourceLinks":45,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/eead71e400420a50f101f88d99a6b4bf5fcfcdadda4e572741bb9b84916f84f0/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-can-llms-improve-their-own-harness-a-controlled-eead71e4","formats":{"html":"/readings/arxiv-cs-ai-can-llms-improve-their-own-harness-a-controlled-eead71e4","markdown":"/readings/arxiv-cs-ai-can-llms-improve-their-own-harness-a-controlled-eead71e4.md","json":"/readings/arxiv-cs-ai-can-llms-improve-their-own-harness-a-controlled-eead71e4.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"},{"id":"term-frontier-model","name":"Frontier model","explanation":"One of the most capable AI models available at a given time; the set shifts as models improve.","referenceUrl":"https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper"},{"id":"term-tool-use","name":"Tool use","explanation":"A model's ability to operate other software, such as a search engine, a calculator or a code runner, while it works.","referenceUrl":"https://docs.anthropic.com/en/docs/build-with-claude/tool-use/overview"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"eead71e400420a50f101f88d99a6b4bf5fcfcdadda4e572741bb9b84916f84f0","revision":"fdcfb83b-921b-4a5b-8078-803fb93907b9","revisionKind":"publication-id","title":"arXiv cs.AI: Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3","url":"https://escapemeridian.com/readings/arxiv-cs-ai-can-llms-improve-their-own-harness-a-controlled-eead71e4"},"source":{"type":"source","id":"https://arxiv.org/abs/2610.30499","revision":"d9fd0c4cb9d3db238cdc34eff875ab5cbb6ccf73a5257c2543ff10885f0e39d8","revisionKind":"sha256-source-text","title":"Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3","url":"https://arxiv.org/abs/2610.30499"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"eead71e400420a50f101f88d99a6b4bf5fcfcdadda4e572741bb9b84916f84f0","title":"arXiv cs.AI: Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3","argument":"In a paper on arXiv cs.AI, Jens Lehmann and Sahar Vahdati study whether frontier models can build and improve their own harness: a symbolic program that coordinates a model’s perception, memory, tool use, recovery, and inference allocation. In their ARC-AGI-3 experiment, each model built a task harness, tried it on five development games, and revised its codebase twice; the researchers evaluated every version on twenty games that did not guide adaptation. Across 27 lineages, performance ranged from zero to near-ceiling scores, but two revisions raised mean held-out RHAE by only 2.80 points, with a 95% interval from -1.44 to 7.35. The authors say development gains overstated improvement on held-out games, and lower inference cost could come at the expense of task performance. They also report that no revision retained its parent. A post hoc rule for selecting among existing snapshots by development score raised the held-out gain to 4.53 points, which the authors frame as a distinction between generating revisions and deciding which to deploy. They conclude that, under this two-revision protocol, strong task performance does not ensure that later revisions generalize or preserve existing capabilities. The abstract notes that human-steered systems show further headroom; it does not quantify that headroom.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The results concern ARC-AGI-3 and a protocol in which models revised the codebase twice; the abstract does not establish how results would change under other tasks or revision protocols. The abstract reports that human-steered systems demonstrate further headroom but does not quantify that headroom. The reported figures concern the study’s RHAE measure and inference cost.","speaker":{"name":"Lehmann, Jens, Vahdati, Sahar","roleAtSource":"Article author"},"source":{"title":"Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.30499","publishedAt":"2026-10-21","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors define a harness as a symbolic program that coordinates several aspects of a language model’s operation.","excerpt":"A harness is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model's behalf.","url":"https://arxiv.org/abs/2610.30499","relationship":"Source-stated"},{"text":"The experiment had each model build a harness, act on five development games, and revise the codebase twice; every version was evaluated on twenty games that did not influence adaptation.","excerpt":"acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation.","url":"https://arxiv.org/abs/2610.30499","relationship":"Source-stated"},{"text":"Across 27 lineages, two revisions raised mean held-out RHAE by 2.80 points, with a 95% interval from -1.44 to 7.35.","excerpt":"Across 27 lineages, harness performance ranges from zero to near-ceiling scores. Two revisions raise mean held-out RHAE by only 2.80 points (95% interval [-1.44,7.35]).","url":"https://arxiv.org/abs/2610.30499","relationship":"Source-stated"},{"text":"The authors report that development gains overstated held-out improvement and that lower inference cost could come at the expense of task performance.","excerpt":"Development gains overstate held-out improvement, and reductions in inference cost can come at the expense of task performance.","url":"https://arxiv.org/abs/2610.30499","relationship":"Source-stated"},{"text":"No revision retained its parent, while a post hoc selection rule based on development score raised the held-out gain to 4.53 points.","excerpt":"No revision retains its parent, yet a post hoc rule that selects among existing snapshots by development score raises the held-out gain to 4.53 points ([0.81,8.68])","url":"https://arxiv.org/abs/2610.30499","relationship":"Source-stated"},{"text":"The authors identify a gap between autonomous harness construction and reliable adaptation under the specific two-revision protocol studied.","excerpt":"These findings reveal a gap between autonomous harness construction and reliable adaptation under this two-revision protocol","url":"https://arxiv.org/abs/2610.30499","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.30499","checkedAt":"2026-11-16T08:52:06.834Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Would additional revisions improve held-out RHAE while preserving existing capabilities under this protocol?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3","summary":"In a paper on arXiv cs.AI, Jens Lehmann and Sahar Vahdati study whether frontier models can build and improve their own harness: a symbolic program that coordinates a model’s perception, memory, tool use, recovery, and inference allocation. In their ARC-AGI-3 experiment, each model built a task harness, tried it on five development games, and revised its codebase twice; the researchers evaluated every version on twenty games that did not guide adaptation. Across 27 lineages, performance ranged from zero to near-ceiling scores, but two revisions raised mean held-out RHAE by only 2.80 points, with a 95% interval from -1.44 to 7.35. The authors say development gains overstated improvement on held-out games, and lower inference cost could come at the expense of task performance. They also report that no revision retained its parent. A post hoc rule for selecting among existing snapshots by development score raised the held-out gain to 4.53 points, which the authors frame as a distinction between generating revisions and deciding which to deploy. They conclude that, under this two-revision protocol, strong task performance does not ensure that later revisions generalize or preserve existing capabilities. The abstract notes that human-steered systems show further headroom; it does not quantify that headroom.","question":"Would additional revisions improve held-out RHAE while preserving existing capabilities under this protocol?","claims":[{"text":"The authors define a harness as a symbolic program that coordinates several aspects of a language model’s operation.","excerpt":"A harness is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model's behalf.","support":"source-stated"},{"text":"The experiment had each model build a harness, act on five development games, and revise the codebase twice; every version was evaluated on twenty games that did not influence adaptation.","excerpt":"acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation.","support":"source-stated"},{"text":"Across 27 lineages, two revisions raised mean held-out RHAE by 2.80 points, with a 95% interval from -1.44 to 7.35.","excerpt":"Across 27 lineages, harness performance ranges from zero to near-ceiling scores. Two revisions raise mean held-out RHAE by only 2.80 points (95% interval [-1.44,7.35]).","support":"source-stated"},{"text":"The authors report that development gains overstated held-out improvement and that lower inference cost could come at the expense of task performance.","excerpt":"Development gains overstate held-out improvement, and reductions in inference cost can come at the expense of task performance.","support":"source-stated"},{"text":"No revision retained its parent, while a post hoc selection rule based on development score raised the held-out gain to 4.53 points.","excerpt":"No revision retains its parent, yet a post hoc rule that selects among existing snapshots by development score raises the held-out gain to 4.53 points ([0.81,8.68])","support":"source-stated"},{"text":"The authors identify a gap between autonomous harness construction and reliable adaptation under the specific two-revision protocol studied.","excerpt":"These findings reveal a gap between autonomous harness construction and reliable adaptation under this two-revision protocol","support":"source-stated"}],"limitations":["The results concern ARC-AGI-3 and a protocol in which models revised the codebase twice; the abstract does not establish how results would change under other tasks or revision protocols.","The abstract reports that human-steered systems demonstrate further headroom but does not quantify that headroom. The reported figures concern the study’s RHAE measure and inference cost."],"mediaId":"f00330389119ea4b3639d2d143b9de933a62cf78f781fc89a17fec8a1a785a2c","correctionReason":null,"id":"eead71e400420a50f101f88d99a6b4bf5fcfcdadda4e572741bb9b84916f84f0","publicationId":"fdcfb83b-921b-4a5b-8078-803fb93907b9","version":1,"sourceUrl":"https://arxiv.org/abs/2610.30499","sourceTitle":"Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-21","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-23T06:25:30.514Z","createdAt":"2026-11-16T07:48:46.801Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"f00330389119ea4b3639d2d143b9de933a62cf78f781fc89a17fec8a1a785a2c","kind":"article","url":"https://arxiv.org/abs/2610.30499","sourceUrl":"https://arxiv.org/abs/2610.30499","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-16T07:47:48.865Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.30499","checkedAt":"2026-11-16T08:52:06.834Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"f00330389119ea4b3639d2d143b9de933a62cf78f781fc89a17fec8a1a785a2c","confidence":0.96,"provider":"typesafe","model":"jev-1.13.0","inputHash":"79aeed8ba40a48b9dbc481f1eace310305dc92a9d89ae20cd2404220cbaccfef","decidedAt":"2026-11-16T07:47:53.716Z"},"provenance":{"textHash":"d9fd0c4cb9d3db238cdc34eff875ab5cbb6ccf73a5257c2543ff10885f0e39d8","observationId":"198a8a2c3f86a3cadcf04aaf7db1b501ad09863ba073dab0736a7dd50d428c79","retrievedAt":"2026-11-16T07:47:48.555Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"78eac4f8-23a9-4578-8cea-55d206496e2a"},"reviewer":{"model":"gpt-6-luna","runId":"68d57e78-7cf6-4535-8e20-a1172f2f03ee"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-16T07:48:46.801Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Lehmann, Jens","url":null,"kind":"unknown"},{"name":"Vahdati, Sahar","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Lehmann, Jens","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Lehmann, Jens","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.30499","locator":"head/meta[name=author|citation_author]"},"assertionId":"b29130cac6e029e0d1929384195cab8813cbfaa8cc6b4944ff0a28b24b4c58cf:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-16T08:52:06.834Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b29130cac6e029e0d1929384195cab8813cbfaa8cc6b4944ff0a28b24b4c58cf","sourceUrl":"https://arxiv.org/abs/2610.30499","contentSha256":"471ab0a08a2561486599879c11699b93ea860eb68747fe2ea2199df7e55d90e5","textHash":"d9fd0c4cb9d3db238cdc34eff875ab5cbb6ccf73a5257c2543ff10885f0e39d8","observedAt":"2026-11-16T08:52:06.834Z"}},{"actor":{"name":"Vahdati, Sahar","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Vahdati, Sahar","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.30499","locator":"head/meta[name=author|citation_author]"},"assertionId":"b29130cac6e029e0d1929384195cab8813cbfaa8cc6b4944ff0a28b24b4c58cf:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-16T08:52:06.834Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b29130cac6e029e0d1929384195cab8813cbfaa8cc6b4944ff0a28b24b4c58cf","sourceUrl":"https://arxiv.org/abs/2610.30499","contentSha256":"471ab0a08a2561486599879c11699b93ea860eb68747fe2ea2199df7e55d90e5","textHash":"d9fd0c4cb9d3db238cdc34eff875ab5cbb6ccf73a5257c2543ff10885f0e39d8","observedAt":"2026-11-16T08:52:06.834Z"}}],"participationCoverage":{"total":2,"returned":2,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.30499","snapshotId":"b29130cac6e029e0d1929384195cab8813cbfaa8cc6b4944ff0a28b24b4c58cf","contentSha256":"471ab0a08a2561486599879c11699b93ea860eb68747fe2ea2199df7e55d90e5","textHash":"d9fd0c4cb9d3db238cdc34eff875ab5cbb6ccf73a5257c2543ff10885f0e39d8","observedAt":"2026-11-16T08:52:06.834Z","basis":"publisher-page-metadata","sourceLinks":45,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/eead71e400420a50f101f88d99a6b4bf5fcfcdadda4e572741bb9b84916f84f0/source"}}}],"audits":[{"publicationId":"fdcfb83b-921b-4a5b-8078-803fb93907b9","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the study to Jens Lehmann and Sahar Vahdati in a paper on arXiv cs.AI. This matches the author list and listing. Findings are kept as the authors' reports rather than established facts.","excerpt":"Authors:Jens Lehmann, Sahar Vahdati"},{"dimension":"summary","status":"uncertain","explanation":"The summary is faithful but hard for non-specialists to follow. Three central terms go unexplained: RHAE (the source expands it as Relative Human Action Efficiency), ARC-AGI-3 (an interactive benchmark where agents explore unfamiliar environments and infer goals), and 'no revision retained its parent'. Two pieces of significance are also dropped: the 4.53 gain's interval [0.81,8.68], which excludes zero while the 2.80 interval does not, and the context that human-steered systems already reach ceiling scores. Smallest repair: briefly define these terms and add that interval and context.","excerpt":"ARC-AGI-3 is an interactive benchmark in which agents must explore unfamiliar environments, figure out their goals on the fly, and keep learning as they go."},{"dimension":"date","status":"pass","explanation":"originalPublished is 2026-10-21. This matches the v1 submission date, not the later v2 revision of 13 Nov 2026 or the feed date.","excerpt":"[Submitted on 21 Oct 2026 (v1), last revised 13 Nov 2026 (this version, v2)]"},{"dimension":"support","status":"pass","explanation":"Each claim maps to verbatim abstract text: the harness definition, the protocol, the 2.80-point gain and its interval, development-versus-held-out overstatement, the cost trade-off, the 4.53-point post hoc gain, and the gap under the two-revision protocol. The limitations and the question stay conditional and introduce no unsupported benefits or costs.","excerpt":"Two revisions raise mean held-out RHAE by only 2.80 points (95% interval [-1.44,7.35])."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the arXiv abstract page. That page is the original source.","excerpt":null}],"createdAt":"2026-11-16T07:48:46.801Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}