{"id":"a5f473a3f984bafdb8f261508ba679f7200a423d4901dd5bd94ed6cbd5fb0707","title":"arXiv cs.AI: Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop","argument":"Zerui Yang and coauthors study when coding agents can stop working after their own tests pass. Their anytime evaluation replays recorded runs, scores intermediate patches with hidden tests, and tracks the cost at each step. Across about 6,000 runs of eight open models in two scaffolds on SWE-bench Verified, the value of continued work varied: the hindsight saving available when self-tests first pass ranged from under 1% to 57%, depending on the model and scaffold. A rule that stops after the same patch passes twice saved 23% in a preregistered replication, but its quality test did not establish non-inferiority—that is, it did not establish that patch quality was not worse. Fine-tuning agents to imitate the rule saved 30% over six seeds, while quality preservation remained uncertain. The authors explain differences between stopping rules by tracking how often each rule fires, whether the patch changes afterward, and whether later changes repair or break it. Their results point to a tradeoff: stopping earlier can save cost, but the evidence leaves open whether quality is preserved.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract reports results for eight open models in two scaffolds on SWE-bench Verified; it does not establish that the findings apply beyond that evaluation setting. The authors state that the preregistered quality test does not establish non-inferiority and that quality preservation remains uncertain. The evaluation compares stopping or selecting an earlier patch without generating new trajectories, provided replay preserves the relevant states.","speaker":{"name":"Yang, Zerui, Xue, Quan, Dong, Miaomiao, Zhang, Chaorui, Zheng, Xuanyu, Wang, Xiyuan, Ruan, Haolin, Niu, Xueyan, Song, Guangcheng, Zhang, Weixi, Han, Wei, Bai, Bo","roleAtSource":"Article author"},"source":{"title":"Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.22901","publishedAt":"2026-11-17","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors evaluate stopping decisions by replaying recorded coding-agent runs, scoring intermediate patches with hidden tests, and measuring cost at each step.","excerpt":"replaying recorded runs, scoring intermediate patches with hidden tests, and measuring the cost incurred at each step","url":"https://arxiv.org/abs/2611.22901","relationship":"Source-stated"},{"text":"Across about 6,000 runs, the hindsight saving available when self-tests pass ranges from under 1% to 57%, depending on the model and scaffold.","excerpt":"The hindsight saving available at passing self-tests ranges from under 1% to 57%, depending on the model and scaffold.","url":"https://arxiv.org/abs/2611.22901","relationship":"Source-stated"},{"text":"A rule that stops after the same patch passes twice saves 23% in a preregistered replication, but the quality test does not establish non-inferiority.","excerpt":"A rule that stops after the same patch passes twice saves 23% in a preregistered replication, but the preregistered quality test does not establish non-inferiority.","url":"https://arxiv.org/abs/2611.22901","relationship":"Source-stated"},{"text":"Fine-tuning agents to imitate the stopping rule saves 30% over six seeds, while quality preservation remains uncertain.","excerpt":"Fine-tuning agents to imitate the rule saves 30% over six seeds, while quality preservation remains uncertain.","url":"https://arxiv.org/abs/2611.22901","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.22901","checkedAt":"2026-11-18T19:50:10.739Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"What evidence would establish whether stopping earlier preserves patch quality across the models and scaffolds studied?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop","summary":"Zerui Yang and coauthors study when coding agents can stop working after their own tests pass. Their anytime evaluation replays recorded runs, scores intermediate patches with hidden tests, and tracks the cost at each step. Across about 6,000 runs of eight open models in two scaffolds on SWE-bench Verified, the value of continued work varied: the hindsight saving available when self-tests first pass ranged from under 1% to 57%, depending on the model and scaffold. A rule that stops after the same patch passes twice saved 23% in a preregistered replication, but its quality test did not establish non-inferiority—that is, it did not establish that patch quality was not worse. Fine-tuning agents to imitate the rule saved 30% over six seeds, while quality preservation remained uncertain. The authors explain differences between stopping rules by tracking how often each rule fires, whether the patch changes afterward, and whether later changes repair or break it. Their results point to a tradeoff: stopping earlier can save cost, but the evidence leaves open whether quality is preserved.","question":"What evidence would establish whether stopping earlier preserves patch quality across the models and scaffolds studied?","claims":[{"text":"The authors evaluate stopping decisions by replaying recorded coding-agent runs, scoring intermediate patches with hidden tests, and measuring cost at each step.","excerpt":"replaying recorded runs, scoring intermediate patches with hidden tests, and measuring the cost incurred at each step","support":"source-stated"},{"text":"Across about 6,000 runs, the hindsight saving available when self-tests pass ranges from under 1% to 57%, depending on the model and scaffold.","excerpt":"The hindsight saving available at passing self-tests ranges from under 1% to 57%, depending on the model and scaffold.","support":"source-stated"},{"text":"A rule that stops after the same patch passes twice saves 23% in a preregistered replication, but the quality test does not establish non-inferiority.","excerpt":"A rule that stops after the same patch passes twice saves 23% in a preregistered replication, but the preregistered quality test does not establish non-inferiority.","support":"source-stated"},{"text":"Fine-tuning agents to imitate the stopping rule saves 30% over six seeds, while quality preservation remains uncertain.","excerpt":"Fine-tuning agents to imitate the rule saves 30% over six seeds, while quality preservation remains uncertain.","support":"source-stated"}],"limitations":["The abstract reports results for eight open models in two scaffolds on SWE-bench Verified; it does not establish that the findings apply beyond that evaluation setting.","The authors state that the preregistered quality test does not establish non-inferiority and that quality preservation remains uncertain.","The evaluation compares stopping or selecting an earlier patch without generating new trajectories, provided replay preserves the relevant states."],"mediaId":"78ff58f3387630eae9d9637f9f55c0747af63cab460b942f3cec7ad83de88091","correctionReason":null,"id":"a5f473a3f984bafdb8f261508ba679f7200a423d4901dd5bd94ed6cbd5fb0707","publicationId":"b6cc402f-4eef-4a70-8d71-7dbdb7f4de59","version":1,"sourceUrl":"https://arxiv.org/abs/2611.22901","sourceTitle":"Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-17","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-18T05:04:04.042Z","createdAt":"2026-11-18T18:47:31.854Z","historical":false,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"78ff58f3387630eae9d9637f9f55c0747af63cab460b942f3cec7ad83de88091","kind":"article","url":"https://arxiv.org/abs/2611.22901","sourceUrl":"https://arxiv.org/abs/2611.22901","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-18T18:34:43.856Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.22901","checkedAt":"2026-11-18T19:50:10.739Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"78ff58f3387630eae9d9637f9f55c0747af63cab460b942f3cec7ad83de88091","confidence":0.82,"provider":"typesafe","model":"jev-1.13.0","inputHash":"eee6945babe24c296f983425e3e69b8d489c307464babf4af8efe28101ffe2b2","decidedAt":"2026-11-18T18:34:44.999Z"},"provenance":{"textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observationId":"06027560994d4430d1baaeb59593458a74cba8da50b8ea717f4f44b6d4b3d804","retrievedAt":"2026-11-18T18:34:43.688Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"507c8360-c726-4502-8c68-9ab1bad25ad3"},"reviewer":{"model":"gpt-6-luna","runId":"3a47de06-311a-4f16-89b4-7c53e13d725c"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-18T18:47:31.854Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Yang, Zerui","url":null,"kind":"unknown"},{"name":"Xue, Quan","url":null,"kind":"unknown"},{"name":"Dong, Miaomiao","url":null,"kind":"unknown"},{"name":"Zhang, Chaorui","url":null,"kind":"unknown"},{"name":"Zheng, Xuanyu","url":null,"kind":"unknown"},{"name":"Wang, Xiyuan","url":null,"kind":"unknown"},{"name":"Ruan, Haolin","url":null,"kind":"unknown"},{"name":"Niu, Xueyan","url":null,"kind":"unknown"},{"name":"Song, Guangcheng","url":null,"kind":"unknown"},{"name":"Zhang, Weixi","url":null,"kind":"unknown"},{"name":"Han, Wei","url":null,"kind":"unknown"},{"name":"Bai, Bo","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Yang, Zerui","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yang, Zerui","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Xue, Quan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Xue, Quan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Dong, Miaomiao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Dong, Miaomiao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Zhang, Chaorui","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Chaorui","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Zheng, Xuanyu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zheng, Xuanyu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Wang, Xiyuan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wang, Xiyuan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Ruan, Haolin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ruan, Haolin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Niu, Xueyan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Niu, Xueyan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Song, Guangcheng","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Song, Guangcheng","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:8","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Zhang, Weixi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Weixi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:9","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Han, Wei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Han, Wei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:10","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Bai, Bo","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Bai, Bo","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:11","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}}],"participationCoverage":{"total":12,"returned":12,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.22901","snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z","basis":"publisher-page-metadata","sourceLinks":50,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/a5f473a3f984bafdb8f261508ba679f7200a423d4901dd5bd94ed6cbd5fb0707/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-done-but-still-working-anytime-evaluation-of-when-a5f473a3","formats":{"html":"/readings/arxiv-cs-ai-done-but-still-working-anytime-evaluation-of-when-a5f473a3","markdown":"/readings/arxiv-cs-ai-done-but-still-working-anytime-evaluation-of-when-a5f473a3.md","json":"/readings/arxiv-cs-ai-done-but-still-working-anytime-evaluation-of-when-a5f473a3.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-fine-tuning","name":"Fine-tuning","explanation":"Further training of an existing model to adapt it to a particular task or use.","referenceUrl":"https://developers.google.com/machine-learning/glossary#fine-tuning"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"a5f473a3f984bafdb8f261508ba679f7200a423d4901dd5bd94ed6cbd5fb0707","revision":"b6cc402f-4eef-4a70-8d71-7dbdb7f4de59","revisionKind":"publication-id","title":"arXiv cs.AI: Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop","url":"https://escapemeridian.com/readings/arxiv-cs-ai-done-but-still-working-anytime-evaluation-of-when-a5f473a3"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.22901","revision":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","revisionKind":"sha256-source-text","title":"Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop","url":"https://arxiv.org/abs/2611.22901"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"a5f473a3f984bafdb8f261508ba679f7200a423d4901dd5bd94ed6cbd5fb0707","title":"arXiv cs.AI: Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop","argument":"Zerui Yang and coauthors study when coding agents can stop working after their own tests pass. Their anytime evaluation replays recorded runs, scores intermediate patches with hidden tests, and tracks the cost at each step. Across about 6,000 runs of eight open models in two scaffolds on SWE-bench Verified, the value of continued work varied: the hindsight saving available when self-tests first pass ranged from under 1% to 57%, depending on the model and scaffold. A rule that stops after the same patch passes twice saved 23% in a preregistered replication, but its quality test did not establish non-inferiority—that is, it did not establish that patch quality was not worse. Fine-tuning agents to imitate the rule saved 30% over six seeds, while quality preservation remained uncertain. The authors explain differences between stopping rules by tracking how often each rule fires, whether the patch changes afterward, and whether later changes repair or break it. Their results point to a tradeoff: stopping earlier can save cost, but the evidence leaves open whether quality is preserved.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract reports results for eight open models in two scaffolds on SWE-bench Verified; it does not establish that the findings apply beyond that evaluation setting. The authors state that the preregistered quality test does not establish non-inferiority and that quality preservation remains uncertain. The evaluation compares stopping or selecting an earlier patch without generating new trajectories, provided replay preserves the relevant states.","speaker":{"name":"Yang, Zerui, Xue, Quan, Dong, Miaomiao, Zhang, Chaorui, Zheng, Xuanyu, Wang, Xiyuan, Ruan, Haolin, Niu, Xueyan, Song, Guangcheng, Zhang, Weixi, Han, Wei, Bai, Bo","roleAtSource":"Article author"},"source":{"title":"Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.22901","publishedAt":"2026-11-17","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors evaluate stopping decisions by replaying recorded coding-agent runs, scoring intermediate patches with hidden tests, and measuring cost at each step.","excerpt":"replaying recorded runs, scoring intermediate patches with hidden tests, and measuring the cost incurred at each step","url":"https://arxiv.org/abs/2611.22901","relationship":"Source-stated"},{"text":"Across about 6,000 runs, the hindsight saving available when self-tests pass ranges from under 1% to 57%, depending on the model and scaffold.","excerpt":"The hindsight saving available at passing self-tests ranges from under 1% to 57%, depending on the model and scaffold.","url":"https://arxiv.org/abs/2611.22901","relationship":"Source-stated"},{"text":"A rule that stops after the same patch passes twice saves 23% in a preregistered replication, but the quality test does not establish non-inferiority.","excerpt":"A rule that stops after the same patch passes twice saves 23% in a preregistered replication, but the preregistered quality test does not establish non-inferiority.","url":"https://arxiv.org/abs/2611.22901","relationship":"Source-stated"},{"text":"Fine-tuning agents to imitate the stopping rule saves 30% over six seeds, while quality preservation remains uncertain.","excerpt":"Fine-tuning agents to imitate the rule saves 30% over six seeds, while quality preservation remains uncertain.","url":"https://arxiv.org/abs/2611.22901","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.22901","checkedAt":"2026-11-18T19:50:10.739Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"What evidence would establish whether stopping earlier preserves patch quality across the models and scaffolds studied?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop","summary":"Zerui Yang and coauthors study when coding agents can stop working after their own tests pass. Their anytime evaluation replays recorded runs, scores intermediate patches with hidden tests, and tracks the cost at each step. Across about 6,000 runs of eight open models in two scaffolds on SWE-bench Verified, the value of continued work varied: the hindsight saving available when self-tests first pass ranged from under 1% to 57%, depending on the model and scaffold. A rule that stops after the same patch passes twice saved 23% in a preregistered replication, but its quality test did not establish non-inferiority—that is, it did not establish that patch quality was not worse. Fine-tuning agents to imitate the rule saved 30% over six seeds, while quality preservation remained uncertain. The authors explain differences between stopping rules by tracking how often each rule fires, whether the patch changes afterward, and whether later changes repair or break it. Their results point to a tradeoff: stopping earlier can save cost, but the evidence leaves open whether quality is preserved.","question":"What evidence would establish whether stopping earlier preserves patch quality across the models and scaffolds studied?","claims":[{"text":"The authors evaluate stopping decisions by replaying recorded coding-agent runs, scoring intermediate patches with hidden tests, and measuring cost at each step.","excerpt":"replaying recorded runs, scoring intermediate patches with hidden tests, and measuring the cost incurred at each step","support":"source-stated"},{"text":"Across about 6,000 runs, the hindsight saving available when self-tests pass ranges from under 1% to 57%, depending on the model and scaffold.","excerpt":"The hindsight saving available at passing self-tests ranges from under 1% to 57%, depending on the model and scaffold.","support":"source-stated"},{"text":"A rule that stops after the same patch passes twice saves 23% in a preregistered replication, but the quality test does not establish non-inferiority.","excerpt":"A rule that stops after the same patch passes twice saves 23% in a preregistered replication, but the preregistered quality test does not establish non-inferiority.","support":"source-stated"},{"text":"Fine-tuning agents to imitate the stopping rule saves 30% over six seeds, while quality preservation remains uncertain.","excerpt":"Fine-tuning agents to imitate the rule saves 30% over six seeds, while quality preservation remains uncertain.","support":"source-stated"}],"limitations":["The abstract reports results for eight open models in two scaffolds on SWE-bench Verified; it does not establish that the findings apply beyond that evaluation setting.","The authors state that the preregistered quality test does not establish non-inferiority and that quality preservation remains uncertain.","The evaluation compares stopping or selecting an earlier patch without generating new trajectories, provided replay preserves the relevant states."],"mediaId":"78ff58f3387630eae9d9637f9f55c0747af63cab460b942f3cec7ad83de88091","correctionReason":null,"id":"a5f473a3f984bafdb8f261508ba679f7200a423d4901dd5bd94ed6cbd5fb0707","publicationId":"b6cc402f-4eef-4a70-8d71-7dbdb7f4de59","version":1,"sourceUrl":"https://arxiv.org/abs/2611.22901","sourceTitle":"Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-17","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-18T05:04:04.042Z","createdAt":"2026-11-18T18:47:31.854Z","historical":false,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"78ff58f3387630eae9d9637f9f55c0747af63cab460b942f3cec7ad83de88091","kind":"article","url":"https://arxiv.org/abs/2611.22901","sourceUrl":"https://arxiv.org/abs/2611.22901","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-18T18:34:43.856Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.22901","checkedAt":"2026-11-18T19:50:10.739Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"78ff58f3387630eae9d9637f9f55c0747af63cab460b942f3cec7ad83de88091","confidence":0.82,"provider":"typesafe","model":"jev-1.13.0","inputHash":"eee6945babe24c296f983425e3e69b8d489c307464babf4af8efe28101ffe2b2","decidedAt":"2026-11-18T18:34:44.999Z"},"provenance":{"textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observationId":"06027560994d4430d1baaeb59593458a74cba8da50b8ea717f4f44b6d4b3d804","retrievedAt":"2026-11-18T18:34:43.688Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"507c8360-c726-4502-8c68-9ab1bad25ad3"},"reviewer":{"model":"gpt-6-luna","runId":"3a47de06-311a-4f16-89b4-7c53e13d725c"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-18T18:47:31.854Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Yang, Zerui","url":null,"kind":"unknown"},{"name":"Xue, Quan","url":null,"kind":"unknown"},{"name":"Dong, Miaomiao","url":null,"kind":"unknown"},{"name":"Zhang, Chaorui","url":null,"kind":"unknown"},{"name":"Zheng, Xuanyu","url":null,"kind":"unknown"},{"name":"Wang, Xiyuan","url":null,"kind":"unknown"},{"name":"Ruan, Haolin","url":null,"kind":"unknown"},{"name":"Niu, Xueyan","url":null,"kind":"unknown"},{"name":"Song, Guangcheng","url":null,"kind":"unknown"},{"name":"Zhang, Weixi","url":null,"kind":"unknown"},{"name":"Han, Wei","url":null,"kind":"unknown"},{"name":"Bai, Bo","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Yang, Zerui","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yang, Zerui","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Xue, Quan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Xue, Quan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Dong, Miaomiao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Dong, Miaomiao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Zhang, Chaorui","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Chaorui","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Zheng, Xuanyu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zheng, Xuanyu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Wang, Xiyuan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wang, Xiyuan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Ruan, Haolin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ruan, Haolin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Niu, Xueyan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Niu, Xueyan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Song, Guangcheng","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Song, Guangcheng","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:8","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Zhang, Weixi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Weixi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:9","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Han, Wei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Han, Wei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:10","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}},{"actor":{"name":"Bai, Bo","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Bai, Bo","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.22901","locator":"head/meta[name=author|citation_author]"},"assertionId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862:participation:11","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T19:50:10.739Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","sourceUrl":"https://arxiv.org/abs/2611.22901","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z"}}],"participationCoverage":{"total":12,"returned":12,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.22901","snapshotId":"3228917459a08354aaffa8020fa66a1fc65a72d5b97f741bcda0ceefceea3862","contentSha256":"b3c1dbac9b158a2b5a6b67795365770fe8c307dc57e0222f0fcb9e33b1fa1797","textHash":"f50196e5f16b70a9f92f687c163841f1b4e8a1591d04c1dc42ec843991256e22","observedAt":"2026-11-18T19:50:10.739Z","basis":"publisher-page-metadata","sourceLinks":50,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/a5f473a3f984bafdb8f261508ba679f7200a423d4901dd5bd94ed6cbd5fb0707/source"}}}],"audits":[{"publicationId":"b6cc402f-4eef-4a70-8d71-7dbdb7f4de59","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The draft correctly credits the arXiv paper to Zerui Yang and coauthors. It presents the findings as the authors' results and keeps their stated caveats on quality.","excerpt":"Authors:Zerui Yang, Quan Xue, Miaomiao Dong, Chaorui Zhang, Xuanyu Zheng, Xiyuan Wang, Haolin Ruan, Xueyan Niu, Guangcheng Song, Weixi Zhang, Wei Han, Bo Bai"},{"dimension":"summary","status":"uncertain","explanation":"The summary is mostly faithful but has two small drifts. First, it says the authors explain differences between stopping rules. The abstract instead explains differences in the value of continued work across models and scaffolds, using measures of how often a rule fires. Smallest repair: write 'explain these differences in the value of continued work.' Second, 'when self-tests first pass' adds 'first,' which the source does not say ('at passing self-tests'). Smallest repair: drop 'first.' The title, question and stated tradeoff otherwise fit the source.","excerpt":"We explain these differences by measuring how often a rule fires, how often the patch changes afterwards, and whether those changes repair or break it."},{"dimension":"date","status":"pass","explanation":"originalPublished is 2026-11-17, which matches the arXiv v1 submission date. The feed date of 2026-11-18 is a later announcement, not the original publication.","excerpt":"[Submitted on 17 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"Each claim quotes the abstract verbatim, and the figures are correct: about 6,000 runs, a range from under 1% to 57%, 23%, and 30% over six seeds. The non-inferiority caveat and the uncertainty about quality are preserved. The limitations, including the replay-state condition, are grounded in the source. The question follows from the stated uncertainty and does not invent benefits.","excerpt":"provided replay preserves the relevant states"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified link to the arXiv abstract page itself, which is the direct original source.","excerpt":null}],"createdAt":"2026-11-18T18:47:31.854Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}