Zhou, Jiawei, Peng, Bo, Huang, Yuxuan, Zhang, Ming, Sun, Youbang, Wang, Xiang, Lu, Chaochao · Oct 27, 2026
arXiv: Researchers question whether skill optimizers improve agentic tasks
Markdown · JSON · Original source
Jiawei Zhou and coauthors argue that skill optimizers—systems that rewrite an agent’s natural-language procedures—may select edits that look useful on a small development set without improving performance on new tasks. Across three optimizers, three benchmarks and two models, learned skills succeeded about as often as their hand-written starting versions, while reading about 16% more input tokens. The authors report that selecting the best of several candidates on at most 30 tasks can produce similar apparent gains even when no candidate changed. They recommend accounting for selection effects and for the extra turns an edit adds, rather than judging a skill by success scores or length alone. The abstract does not establish whether these recommendations improve performance in other settings.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
arXiv cs.AI · Article
Source details
Original source date: Oct 27, 2026 (2026-10-27).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-10-29T22:18:14.142Z. First observed: 2026-10-28T08:38:37.797Z. Retrieved: 2026-10-29T22:17:20.461Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The abstract reports results across three optimizers, three agentic benchmarks and two models; it does not establish how well the findings generalize to other settings. The abstract does not report whether the four proposed principles improve results when implemented.
Open question
Would accounting for selection effects and added turns improve skill optimization on tasks beyond the benchmarks studied?
Source-stated
The authors describe skill optimizers as systems that adapt agents by rewriting natural-language procedures and keeping edits that raise success on a small development set.
rewriting its skill, a natural-language procedure in its context, and keeping the edits that raise success on a small dev set
Source-stated
Across three optimizers, three agentic benchmarks and two models, learned skills succeeded about as often as their hand-written seed, averaging 0.1 points higher.
Across three optimizers, three agentic benchmarks and two models, the learned skills succeed about as often as their hand-written seed, 0.1 points higher on average.
Source-stated
The authors report that selection of the best candidate on at most 30 tasks can produce gains of this size even when candidates are unchanged.
keeping the best of several candidates on at most 30 tasks reports gains this large even when none changed.
Source-stated
The authors report that learned skills read about 16% more input tokens and often add rules that prompt extra turns.
They read about 16% more input tokens: the optimizers almost always add rules
Source-stated
The authors recommend testing whether an edit's gain exceeds what selection alone reports, and accounting for added turns.
accept an edit only if its gain exceeds what selection alone reports; charge it for the turns it adds, not for its length
Source and contributor context
Zhou, Jiawei, Peng, Bo, Huang, Yuxuan, Zhang, Ming, Sun, Youbang, Wang, Xiang, Lu, Chaochao · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Agentic AI
AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Model tokens
Small chunks of text, often parts of words, that a language model reads and writes; usage and prices are often counted in tokens.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: a03e5104-bb25-46bb-8e98-38527932b6d0.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-10-29T22:18:14.142Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 601b64da073d9bd683411045fa28e8ffe7c5d780ecbc051102488e960e8528ec. Source text hash: 5cf4c65b7ad2722e6d2f67f0e9f42b58b5b037e5e3fee95eb43a339b49f7defb.
Revision and source-check history
Version 1 · 2026-10-29T22:18:14.142Z
arXiv: Researchers question whether skill optimizers improve agentic tasks
Jiawei Zhou and coauthors argue that skill optimizers—systems that rewrite an agent’s natural-language procedures—may select edits that look useful on a small development set without improving performance on new tasks. Across three optimizers, three benchmarks and two models, learned skills succeeded about as often as their hand-written starting versions, while reading about 16% more input tokens. The authors report that selecting the best of several candidates on at most 30 tasks can produce similar apparent gains even when no candidate changed. They recommend accounting for selection effects and for the extra turns an edit adds, rather than judging a skill by success scores or length alone. The abstract does not establish whether these recommendations improve performance in other settings.
Original source
arXiv cs.AI · Article
Source details
Original source date: Oct 27, 2026 (2026-10-27).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-10-29T22:18:14.142Z. First observed: 2026-10-28T08:38:37.797Z. Retrieved: 2026-10-29T22:17:20.461Z.
Limits
The abstract reports results across three optimizers, three agentic benchmarks and two models; it does not establish how well the findings generalize to other settings. The abstract does not report whether the four proposed principles improve results when implemented.
Open question
Would accounting for selection effects and added turns improve skill optimization on tasks beyond the benchmarks studied?
Source-stated
The authors describe skill optimizers as systems that adapt agents by rewriting natural-language procedures and keeping edits that raise success on a small development set.
rewriting its skill, a natural-language procedure in its context, and keeping the edits that raise success on a small dev set
Source-stated
Across three optimizers, three agentic benchmarks and two models, learned skills succeeded about as often as their hand-written seed, averaging 0.1 points higher.
Across three optimizers, three agentic benchmarks and two models, the learned skills succeed about as often as their hand-written seed, 0.1 points higher on average.
Source-stated
The authors report that selection of the best candidate on at most 30 tasks can produce gains of this size even when candidates are unchanged.
keeping the best of several candidates on at most 30 tasks reports gains this large even when none changed.
Source-stated
The authors report that learned skills read about 16% more input tokens and often add rules that prompt extra turns.
They read about 16% more input tokens: the optimizers almost always add rules
Source-stated
The authors recommend testing whether an edit's gain exceeds what selection alone reports, and accounting for added turns.
accept an edit only if its gain exceeds what selection alone reports; charge it for the turns it adds, not for its length
Source and contributor context
Zhou, Jiawei, Peng, Bo, Huang, Yuxuan, Zhang, Ming, Sun, Youbang, Wang, Xiang, Lu, Chaochao · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Agentic AI
AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Model tokens
Small chunks of text, often parts of words, that a language model reads and writes; usage and prices are often counted in tokens.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: a03e5104-bb25-46bb-8e98-38527932b6d0.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-10-29T22:18:14.142Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 601b64da073d9bd683411045fa28e8ffe7c5d780ecbc051102488e960e8528ec. Source text hash: 5cf4c65b7ad2722e6d2f67f0e9f42b58b5b037e5e3fee95eb43a339b49f7defb.
passed · 2026-10-29T22:18:14.142Z · anthropic/claude-opus-5.5
Publication: a03e5104-bb25-46bb-8e98-38527932b6d0
attribution — pass: The draft correctly attributes the paper to Jiawei Zhou and coauthors on arXiv. It frames the findings and recommendations as the authors' reports and arguments.
Authors:Jiawei Zhou, Bo Peng, Yuxuan Huang, Ming Zhang, Youbang Sun, Xiang Wang, Chaochao Lu
summary — pass: The title, summary and question match the abstract. Skill optimizers are explained in plain language, and the central finding is clear: dev-set gains vanish on test and costs rise. The draft notes the limits on generalization, and the question is open rather than leading. Terms such as tokens and turns are only lightly explained, but they do not block understanding.
Their dev gains vanish on test: keeping the best of several candidates on at most 30 tasks reports gains this large even when none changed.
date — pass: The original publication date, 2026-10-27, matches the v1 submission date. The feed date of 2026-10-29 is a later announcement.
[Submitted on 27 Oct 2026]
support — pass: Each claim matches the abstract: the 0.1-point average gain, the 30-task selection effect, the roughly 16% extra input tokens, rules that lead to extra turns, and the recommended principles. The draft states that real-world benefit of the recommendations is unestablished, and it adds no unsupported impacts.
They read about 16% more input tokens: the optimizers almost always add rules, half to re-check work and some to act one call at a time, and the agent follows them as extra turns
media — pass: The selected media item is the verified link to the arXiv abstract page itself, which is the direct original source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.