Yang, Zerui, Xue, Quan, Dong, Miaomiao, Zhang, Chaorui, Zheng, Xuanyu, Wang, Xiyuan, Ruan, Haolin, Niu, Xueyan, Song, Guangcheng, Zhang, Weixi, Han, Wei, Bai, Bo · Nov 17, 2026
arXiv cs.AI: Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop
Markdown · JSON · Original source
Zerui Yang and coauthors study when coding agents can stop working after their own tests pass. Their anytime evaluation replays recorded runs, scores intermediate patches with hidden tests, and tracks the cost at each step. Across about 6,000 runs of eight open models in two scaffolds on SWE-bench Verified, the value of continued work varied: the hindsight saving available when self-tests first pass ranged from under 1% to 57%, depending on the model and scaffold. A rule that stops after the same patch passes twice saved 23% in a preregistered replication, but its quality test did not establish non-inferiority—that is, it did not establish that patch quality was not worse. agents to imitate the rule saved 30% over six seeds, while quality preservation remained uncertain. The authors explain differences between stopping rules by tracking how often each rule fires, whether the patch changes afterward, and whether later changes repair or break it. Their results point to a tradeoff: stopping earlier can save cost, but the evidence leaves open whether quality is preserved.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 17, 2026 (2026-11-17).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-18T18:47:31.854Z. First observed: 2026-11-18T05:04:04.042Z. Retrieved: 2026-11-18T18:34:43.688Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The abstract reports results for eight open models in two scaffolds on SWE-bench Verified; it does not establish that the findings apply beyond that evaluation setting. The authors state that the preregistered quality test does not establish non-inferiority and that quality preservation remains uncertain. The evaluation compares stopping or selecting an earlier patch without generating new trajectories, provided replay preserves the relevant states.
Open question
What evidence would establish whether stopping earlier preserves patch quality across the models and scaffolds studied?
Source-stated
The authors evaluate stopping decisions by replaying recorded coding-agent runs, scoring intermediate patches with hidden tests, and measuring cost at each step.
replaying recorded runs, scoring intermediate patches with hidden tests, and measuring the cost incurred at each step
Source-stated
Across about 6,000 runs, the hindsight saving available when self-tests pass ranges from under 1% to 57%, depending on the model and scaffold.
The hindsight saving available at passing self-tests ranges from under 1% to 57%, depending on the model and scaffold.
Source-stated
A rule that stops after the same patch passes twice saves 23% in a preregistered replication, but the quality test does not establish non-inferiority.
A rule that stops after the same patch passes twice saves 23% in a preregistered replication, but the preregistered quality test does not establish non-inferiority.
Source-stated
Fine-tuning agents to imitate the stopping rule saves 30% over six seeds, while quality preservation remains uncertain.
Fine-tuning agents to imitate the rule saves 30% over six seeds, while quality preservation remains uncertain.
Source and contributor context
Yang, Zerui, Xue, Quan, Dong, Miaomiao, Zhang, Chaorui, Zheng, Xuanyu, Wang, Xiyuan, Ruan, Haolin, Niu, Xueyan, Song, Guangcheng, Zhang, Weixi, Han, Wei, Bai, Bo · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Fine-tuning
Further training of an existing model to adapt it to a particular task or use.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: b6cc402f-4eef-4a70-8d71-7dbdb7f4de59.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T18:47:31.854Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 06027560994d4430d1baaeb59593458a74cba8da50b8ea717f4f44b6d4b3d804. Source text hash: f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22.
Revision and source-check history
Version 1 · 2026-11-18T18:47:31.854Z
arXiv cs.AI: Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop
Zerui Yang and coauthors study when coding agents can stop working after their own tests pass. Their anytime evaluation replays recorded runs, scores intermediate patches with hidden tests, and tracks the cost at each step. Across about 6,000 runs of eight open models in two scaffolds on SWE-bench Verified, the value of continued work varied: the hindsight saving available when self-tests first pass ranged from under 1% to 57%, depending on the model and scaffold. A rule that stops after the same patch passes twice saved 23% in a preregistered replication, but its quality test did not establish non-inferiority—that is, it did not establish that patch quality was not worse. agents to imitate the rule saved 30% over six seeds, while quality preservation remained uncertain. The authors explain differences between stopping rules by tracking how often each rule fires, whether the patch changes afterward, and whether later changes repair or break it. Their results point to a tradeoff: stopping earlier can save cost, but the evidence leaves open whether quality is preserved.
Original source
cs.AI · Article
Source details
Original source date: Nov 17, 2026 (2026-11-17).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-18T18:47:31.854Z. First observed: 2026-11-18T05:04:04.042Z. Retrieved: 2026-11-18T18:34:43.688Z.
Limits
The abstract reports results for eight open models in two scaffolds on SWE-bench Verified; it does not establish that the findings apply beyond that evaluation setting. The authors state that the preregistered quality test does not establish non-inferiority and that quality preservation remains uncertain. The evaluation compares stopping or selecting an earlier patch without generating new trajectories, provided replay preserves the relevant states.
Open question
What evidence would establish whether stopping earlier preserves patch quality across the models and scaffolds studied?
Source-stated
The authors evaluate stopping decisions by replaying recorded coding-agent runs, scoring intermediate patches with hidden tests, and measuring cost at each step.
replaying recorded runs, scoring intermediate patches with hidden tests, and measuring the cost incurred at each step
Source-stated
Across about 6,000 runs, the hindsight saving available when self-tests pass ranges from under 1% to 57%, depending on the model and scaffold.
The hindsight saving available at passing self-tests ranges from under 1% to 57%, depending on the model and scaffold.
Source-stated
A rule that stops after the same patch passes twice saves 23% in a preregistered replication, but the quality test does not establish non-inferiority.
A rule that stops after the same patch passes twice saves 23% in a preregistered replication, but the preregistered quality test does not establish non-inferiority.
Source-stated
Fine-tuning agents to imitate the stopping rule saves 30% over six seeds, while quality preservation remains uncertain.
Fine-tuning agents to imitate the rule saves 30% over six seeds, while quality preservation remains uncertain.
Source and contributor context
Yang, Zerui, Xue, Quan, Dong, Miaomiao, Zhang, Chaorui, Zheng, Xuanyu, Wang, Xiyuan, Ruan, Haolin, Niu, Xueyan, Song, Guangcheng, Zhang, Weixi, Han, Wei, Bai, Bo · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Fine-tuning
Further training of an existing model to adapt it to a particular task or use.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: b6cc402f-4eef-4a70-8d71-7dbdb7f4de59.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T18:47:31.854Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 06027560994d4430d1baaeb59593458a74cba8da50b8ea717f4f44b6d4b3d804. Source text hash: f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22.
uncertain · 2026-11-18T18:47:31.854Z · anthropic/claude-opus-5.5
Publication: b6cc402f-4eef-4a70-8d71-7dbdb7f4de59
attribution — pass: The draft correctly credits the arXiv paper to Zerui Yang and coauthors. It presents the findings as the authors' results and keeps their stated caveats on quality.
Authors:Zerui Yang, Quan Xue, Miaomiao Dong, Chaorui Zhang, Xuanyu Zheng, Xiyuan Wang, Haolin Ruan, Xueyan Niu, Guangcheng Song, Weixi Zhang, Wei Han, Bo Bai
summary — uncertain: The summary is mostly faithful but has two small drifts. First, it says the authors explain differences between stopping rules. The abstract instead explains differences in the value of continued work across models and scaffolds, using measures of how often a rule fires. Smallest repair: write 'explain these differences in the value of continued work.' Second, 'when self-tests first pass' adds 'first,' which the source does not say ('at passing self-tests'). Smallest repair: drop 'first.' The title, question and stated tradeoff otherwise fit the source.
We explain these differences by measuring how often a rule fires, how often the patch changes afterwards, and whether those changes repair or break it.
date — pass: originalPublished is 2026-11-17, which matches the arXiv v1 submission date. The feed date of 2026-11-18 is a later announcement, not the original publication.
[Submitted on 17 Nov 2026]
support — pass: Each claim quotes the abstract verbatim, and the figures are correct: about 6,000 runs, a range from under 1% to 57%, 23%, and 30% over six seeds. The non-inferiority caveat and the uncertainty about quality are preserved. The limitations, including the replay-state condition, are grounded in the source. The question follows from the stated uncertainty and does not invent benefits.
provided replay preserves the relevant states
media — pass: The selected media is the verified link to the arXiv abstract page itself, which is the direct original source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Further training of an existing model to adapt it to a particular task or use.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivFurther training of an existing model to adapt it to a particular task or use.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv