{"id":"a3a4f4a3d8deceaffeb461bb2d13a47a88ead05f3b612b41805e290fc581ebbd","title":"arXiv cs.AI: Buchanan et al. argue benchmarks need continuous maintenance","argument":"E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman and Christopher Ré argue that benchmarks can become stale even after careful validation. As models improve, they may find valid solutions that verifiers reject; changes to evaluation infrastructure can also undermine assumptions built into tasks and tests. The authors call the resulting gap between scores and intended capabilities “benchmark maintenance debt.”\n\nThey describe a continuous-validation workflow combining community reports, task artifacts, agent traces, automated oracle checks and human-gated repairs. Applying it to Terminal-Bench 2.0, they report issues in roughly 30% of the benchmark and release repairs as Terminal-Bench 2.1. The authors say those repairs raised accuracy by over 10 percentage points and moved leaderboard positions by as many as three ranks, while overall rankings remained highly correlated.\n\nTheir argument goes beyond fixing tasks: fixed wall-clock limits can change agent performance, so resource budgets are part of what a benchmark measures and need validation too. They report that extending the time limit closed a 20pp harness gap on GPT 5.4. For GPT 5.4 mini, it reduced a 30 pp gap to 5 pp. The abstract describes these results but does not detail the audit methods or how the workflow generalizes beyond this benchmark.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence is the arXiv abstract; it does not provide the audit methods, task-level findings or details needed to assess how the reported results were obtained. The abstract reports results for Terminal-Bench 2.0 and does not establish how well the workflow generalizes to other benchmarks.","speaker":{"name":"Buchanan, E. Kelly, Zhou, Crystal, Linderman, Scott W., Ré, Christopher","roleAtSource":"Article author"},"source":{"title":"Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.08417","publishedAt":"2026-11-05","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that benchmark defects can cause scores to diverge from the capabilities the benchmarks are intended to measure, and call this benchmark maintenance debt.","excerpt":"We call the resulting accumulation of defects benchmark maintenance debt: a growing gap between benchmark scores and the capabilities they are intended to measure.","url":"https://arxiv.org/abs/2611.08417","relationship":"Source-stated"},{"text":"The authors report that their audit of Terminal-Bench 2.0 found issues in roughly 30% of the benchmark, and that the repairs changed evaluation results.","excerpt":"Our audit identifies issues in roughly 30% of the benchmark, and we release the resulting repairs as Terminal-Bench 2.1.","url":"https://arxiv.org/abs/2611.08417","relationship":"Source-stated"},{"text":"The authors report that repaired evaluation results increased accuracy by over 10 percentage points and moved leaderboard positions by up to three ranks, while overall rankings remained highly correlated.","excerpt":"accuracy increased by over 10 percentage points and leaderboard positions move by up to three ranks, even as the overall ranking remains highly correlated (Spearman rho = 0.958).","url":"https://arxiv.org/abs/2611.08417","relationship":"Source-stated"},{"text":"The authors argue that resource budgets, including fixed wall-clock limits, are part of the measurement specification and should be validated with benchmark tasks.","excerpt":"resource budgets are part of the measurement specification and must be validated alongside the tasks themselves.","url":"https://arxiv.org/abs/2611.08417","relationship":"Source-stated"},{"text":"The authors report that extending the time limit closed a 20pp harness gap on GPT 5.4 and reduced a 30 pp gap on GPT 5.4 mini to 5 pp.","excerpt":"extending the time limit closes a 20pp harness gap on GPT 5.4 and shrinks a 30 pp gap on GPT 5.4 mini to 5 pp.","url":"https://arxiv.org/abs/2611.08417","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.08417","checkedAt":"2026-11-08T22:42:09.312Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How does the authors’ workflow validate resource budgets alongside task artifacts as models and evaluation infrastructure change?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Buchanan et al. argue benchmarks need continuous maintenance","summary":"E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman and Christopher Ré argue that benchmarks can become stale even after careful validation. As models improve, they may find valid solutions that verifiers reject; changes to evaluation infrastructure can also undermine assumptions built into tasks and tests. The authors call the resulting gap between scores and intended capabilities “benchmark maintenance debt.”\n\nThey describe a continuous-validation workflow combining community reports, task artifacts, agent traces, automated oracle checks and human-gated repairs. Applying it to Terminal-Bench 2.0, they report issues in roughly 30% of the benchmark and release repairs as Terminal-Bench 2.1. The authors say those repairs raised accuracy by over 10 percentage points and moved leaderboard positions by as many as three ranks, while overall rankings remained highly correlated.\n\nTheir argument goes beyond fixing tasks: fixed wall-clock limits can change agent performance, so resource budgets are part of what a benchmark measures and need validation too. They report that extending the time limit closed a 20pp harness gap on GPT 5.4. For GPT 5.4 mini, it reduced a 30 pp gap to 5 pp. The abstract describes these results but does not detail the audit methods or how the workflow generalizes beyond this benchmark.","question":"How does the authors’ workflow validate resource budgets alongside task artifacts as models and evaluation infrastructure change?","claims":[{"text":"The authors argue that benchmark defects can cause scores to diverge from the capabilities the benchmarks are intended to measure, and call this benchmark maintenance debt.","excerpt":"We call the resulting accumulation of defects benchmark maintenance debt: a growing gap between benchmark scores and the capabilities they are intended to measure.","support":"source-stated"},{"text":"The authors report that their audit of Terminal-Bench 2.0 found issues in roughly 30% of the benchmark, and that the repairs changed evaluation results.","excerpt":"Our audit identifies issues in roughly 30% of the benchmark, and we release the resulting repairs as Terminal-Bench 2.1.","support":"source-stated"},{"text":"The authors report that repaired evaluation results increased accuracy by over 10 percentage points and moved leaderboard positions by up to three ranks, while overall rankings remained highly correlated.","excerpt":"accuracy increased by over 10 percentage points and leaderboard positions move by up to three ranks, even as the overall ranking remains highly correlated (Spearman rho = 0.958).","support":"source-stated"},{"text":"The authors argue that resource budgets, including fixed wall-clock limits, are part of the measurement specification and should be validated with benchmark tasks.","excerpt":"resource budgets are part of the measurement specification and must be validated alongside the tasks themselves.","support":"source-stated"},{"text":"The authors report that extending the time limit closed a 20pp harness gap on GPT 5.4 and reduced a 30 pp gap on GPT 5.4 mini to 5 pp.","excerpt":"extending the time limit closes a 20pp harness gap on GPT 5.4 and shrinks a 30 pp gap on GPT 5.4 mini to 5 pp.","support":"source-stated"}],"limitations":["The evidence is the arXiv abstract; it does not provide the audit methods, task-level findings or details needed to assess how the reported results were obtained.","The abstract reports results for Terminal-Bench 2.0 and does not establish how well the workflow generalizes to other benchmarks."],"mediaId":"f829c1d2e9578a614af16090fe4c66ea89bab608f7757c3f9ad1d0bce7e7d4a5","correctionReason":null,"id":"a3a4f4a3d8deceaffeb461bb2d13a47a88ead05f3b612b41805e290fc581ebbd","publicationId":"6de9593e-f0d2-411f-8fd6-ea9800209598","version":1,"sourceUrl":"https://arxiv.org/abs/2611.08417","sourceTitle":"Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-05","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:10:38.570Z","createdAt":"2026-11-08T22:11:11.953Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"f829c1d2e9578a614af16090fe4c66ea89bab608f7757c3f9ad1d0bce7e7d4a5","kind":"article","url":"https://arxiv.org/abs/2611.08417","sourceUrl":"https://arxiv.org/abs/2611.08417","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-08T22:10:06.611Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.08417","checkedAt":"2026-11-08T22:42:09.312Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"f829c1d2e9578a614af16090fe4c66ea89bab608f7757c3f9ad1d0bce7e7d4a5","confidence":0.9400000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"c2c72b1aed3fa5636537d559d1ba7e95ce25ed7dbc29ea59c3a6ed0a4b7515e1","decidedAt":"2026-11-08T22:10:10.907Z"},"provenance":{"textHash":"754af07b4e64cf4fb18d52b0519bb5fa99d663a65b50c7852304663f89573951","observationId":"b221a7ac6e9eae299c52c5dd24954461ecd228bbc3095b7f767ed6de909bb208","retrievedAt":"2026-11-08T22:10:06.083Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"fcd1adca-6bf3-4d29-85ea-039aa166ccd9"},"reviewer":{"model":"gpt-6-luna","runId":"ef82e9e0-907a-473f-8748-bf97fb090c23"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-08T22:11:11.953Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Buchanan, E. Kelly","url":null,"kind":"unknown"},{"name":"Zhou, Crystal","url":null,"kind":"unknown"},{"name":"Linderman, Scott W.","url":null,"kind":"unknown"},{"name":"Ré, Christopher","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Buchanan, E. Kelly","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Buchanan, E. Kelly","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.08417","locator":"head/meta[name=author|citation_author]"},"assertionId":"0ab3f7a7783eda80a0df403a50fb0481f2d8fefc57e5b7a08da76fe5b3fff58b:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T22:42:09.312Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"0ab3f7a7783eda80a0df403a50fb0481f2d8fefc57e5b7a08da76fe5b3fff58b","sourceUrl":"https://arxiv.org/abs/2611.08417","contentSha256":"a594919dd0565a9653f70e80ad0649871d8eb6ae05c0cb71d8e6bcca3b667609","textHash":"754af07b4e64cf4fb18d52b0519bb5fa99d663a65b50c7852304663f89573951","observedAt":"2026-11-08T22:42:09.312Z"}},{"actor":{"name":"Zhou, Crystal","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhou, Crystal","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.08417","locator":"head/meta[name=author|citation_author]"},"assertionId":"0ab3f7a7783eda80a0df403a50fb0481f2d8fefc57e5b7a08da76fe5b3fff58b:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T22:42:09.312Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"0ab3f7a7783eda80a0df403a50fb0481f2d8fefc57e5b7a08da76fe5b3fff58b","sourceUrl":"https://arxiv.org/abs/2611.08417","contentSha256":"a594919dd0565a9653f70e80ad0649871d8eb6ae05c0cb71d8e6bcca3b667609","textHash":"754af07b4e64cf4fb18d52b0519bb5fa99d663a65b50c7852304663f89573951","observedAt":"2026-11-08T22:42:09.312Z"}},{"actor":{"name":"Linderman, Scott W.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Linderman, Scott W.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.08417","locator":"head/meta[name=author|citation_author]"},"assertionId":"0ab3f7a7783eda80a0df403a50fb0481f2d8fefc57e5b7a08da76fe5b3fff58b:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T22:42:09.312Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"0ab3f7a7783eda80a0df403a50fb0481f2d8fefc57e5b7a08da76fe5b3fff58b","sourceUrl":"https://arxiv.org/abs/2611.08417","contentSha256":"a594919dd0565a9653f70e80ad0649871d8eb6ae05c0cb71d8e6bcca3b667609","textHash":"754af07b4e64cf4fb18d52b0519bb5fa99d663a65b50c7852304663f89573951","observedAt":"2026-11-08T22:42:09.312Z"}},{"actor":{"name":"Ré, Christopher","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ré, Christopher","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.08417","locator":"head/meta[name=author|citation_author]"},"assertionId":"0ab3f7a7783eda80a0df403a50fb0481f2d8fefc57e5b7a08da76fe5b3fff58b:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T22:42:09.312Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"0ab3f7a7783eda80a0df403a50fb0481f2d8fefc57e5b7a08da76fe5b3fff58b","sourceUrl":"https://arxiv.org/abs/2611.08417","contentSha256":"a594919dd0565a9653f70e80ad0649871d8eb6ae05c0cb71d8e6bcca3b667609","textHash":"754af07b4e64cf4fb18d52b0519bb5fa99d663a65b50c7852304663f89573951","observedAt":"2026-11-08T22:42:09.312Z"}}],"participationCoverage":{"total":4,"returned":4,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.08417","snapshotId":"0ab3f7a7783eda80a0df403a50fb0481f2d8fefc57e5b7a08da76fe5b3fff58b","contentSha256":"a594919dd0565a9653f70e80ad0649871d8eb6ae05c0cb71d8e6bcca3b667609","textHash":"754af07b4e64cf4fb18d52b0519bb5fa99d663a65b50c7852304663f89573951","observedAt":"2026-11-08T22:42:09.312Z","basis":"publisher-page-metadata","sourceLinks":42,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/a3a4f4a3d8deceaffeb461bb2d13a47a88ead05f3b612b41805e290fc581ebbd/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-buchanan-et-al-argue-benchmarks-need-continuous-a3a4f4a3","formats":{"html":"/readings/arxiv-cs-ai-buchanan-et-al-argue-benchmarks-need-continuous-a3a4f4a3","markdown":"/readings/arxiv-cs-ai-buchanan-et-al-argue-benchmarks-need-continuous-a3a4f4a3.md","json":"/readings/arxiv-cs-ai-buchanan-et-al-argue-benchmarks-need-continuous-a3a4f4a3.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"},{"id":"time","name":"TIME","kind":"organization","context":"TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.","referenceUrls":["https://time.com/"],"path":"/entities/time","json":"/entities/time.json","markdown":"/entities/time.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"a3a4f4a3d8deceaffeb461bb2d13a47a88ead05f3b612b41805e290fc581ebbd","revision":"6de9593e-f0d2-411f-8fd6-ea9800209598","revisionKind":"publication-id","title":"arXiv cs.AI: Buchanan et al. argue benchmarks need continuous maintenance","url":"https://escapemeridian.com/readings/arxiv-cs-ai-buchanan-et-al-argue-benchmarks-need-continuous-a3a4f4a3"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.08417","revision":"754af07b4e64cf4fb18d52b0519bb5fa99d663a65b50c7852304663f89573951","revisionKind":"sha256-source-text","title":"Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks","url":"https://arxiv.org/abs/2611.08417"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"a3a4f4a3d8deceaffeb461bb2d13a47a88ead05f3b612b41805e290fc581ebbd","title":"arXiv cs.AI: Buchanan et al. argue benchmarks need continuous maintenance","argument":"E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman and Christopher Ré argue that benchmarks can become stale even after careful validation. As models improve, they may find valid solutions that verifiers reject; changes to evaluation infrastructure can also undermine assumptions built into tasks and tests. The authors call the resulting gap between scores and intended capabilities “benchmark maintenance debt.”\n\nThey describe a continuous-validation workflow combining community reports, task artifacts, agent traces, automated oracle checks and human-gated repairs. Applying it to Terminal-Bench 2.0, they report issues in roughly 30% of the benchmark and release repairs as Terminal-Bench 2.1. The authors say those repairs raised accuracy by over 10 percentage points and moved leaderboard positions by as many as three ranks, while overall rankings remained highly correlated.\n\nTheir argument goes beyond fixing tasks: fixed wall-clock limits can change agent performance, so resource budgets are part of what a benchmark measures and need validation too. They report that extending the time limit closed a 20pp harness gap on GPT 5.4. For GPT 5.4 mini, it reduced a 30 pp gap to 5 pp. The abstract describes these results but does not detail the audit methods or how the workflow generalizes beyond this benchmark.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence is the arXiv abstract; it does not provide the audit methods, task-level findings or details needed to assess how the reported results were obtained. The abstract reports results for Terminal-Bench 2.0 and does not establish how well the workflow generalizes to other benchmarks.","speaker":{"name":"Buchanan, E. Kelly, Zhou, Crystal, Linderman, Scott W., Ré, Christopher","roleAtSource":"Article author"},"source":{"title":"Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.08417","publishedAt":"2026-11-05","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that benchmark defects can cause scores to diverge from the capabilities the benchmarks are intended to measure, and call this benchmark maintenance debt.","excerpt":"We call the resulting accumulation of defects benchmark maintenance debt: a growing gap between benchmark scores and the capabilities they are intended to measure.","url":"https://arxiv.org/abs/2611.08417","relationship":"Source-stated"},{"text":"The authors report that their audit of Terminal-Bench 2.0 found issues in roughly 30% of the benchmark, and that the repairs changed evaluation results.","excerpt":"Our audit identifies issues in roughly 30% of the benchmark, and we release the resulting repairs as Terminal-Bench 2.1.","url":"https://arxiv.org/abs/2611.08417","relationship":"Source-stated"},{"text":"The authors report that repaired evaluation results increased accuracy by over 10 percentage points and moved leaderboard positions by up to three ranks, while overall rankings remained highly correlated.","excerpt":"accuracy increased by over 10 percentage points and leaderboard positions move by up to three ranks, even as the overall ranking remains highly correlated (Spearman rho = 0.958).","url":"https://arxiv.org/abs/2611.08417","relationship":"Source-stated"},{"text":"The authors argue that resource budgets, including fixed wall-clock limits, are part of the measurement specification and should be validated with benchmark tasks.","excerpt":"resource budgets are part of the measurement specification and must be validated alongside the tasks themselves.","url":"https://arxiv.org/abs/2611.08417","relationship":"Source-stated"},{"text":"The authors report that extending the time limit closed a 20pp harness gap on GPT 5.4 and reduced a 30 pp gap on GPT 5.4 mini to 5 pp.","excerpt":"extending the time limit closes a 20pp harness gap on GPT 5.4 and shrinks a 30 pp gap on GPT 5.4 mini to 5 pp.","url":"https://arxiv.org/abs/2611.08417","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.08417","checkedAt":"2026-11-08T22:42:09.312Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How does the authors’ workflow validate resource budgets alongside task artifacts as models and evaluation infrastructure change?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Buchanan et al. argue benchmarks need continuous maintenance","summary":"E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman and Christopher Ré argue that benchmarks can become stale even after careful validation. As models improve, they may find valid solutions that verifiers reject; changes to evaluation infrastructure can also undermine assumptions built into tasks and tests. The authors call the resulting gap between scores and intended capabilities “benchmark maintenance debt.”\n\nThey describe a continuous-validation workflow combining community reports, task artifacts, agent traces, automated oracle checks and human-gated repairs. Applying it to Terminal-Bench 2.0, they report issues in roughly 30% of the benchmark and release repairs as Terminal-Bench 2.1. The authors say those repairs raised accuracy by over 10 percentage points and moved leaderboard positions by as many as three ranks, while overall rankings remained highly correlated.\n\nTheir argument goes beyond fixing tasks: fixed wall-clock limits can change agent performance, so resource budgets are part of what a benchmark measures and need validation too. They report that extending the time limit closed a 20pp harness gap on GPT 5.4. For GPT 5.4 mini, it reduced a 30 pp gap to 5 pp. The abstract describes these results but does not detail the audit methods or how the workflow generalizes beyond this benchmark.","question":"How does the authors’ workflow validate resource budgets alongside task artifacts as models and evaluation infrastructure change?","claims":[{"text":"The authors argue that benchmark defects can cause scores to diverge from the capabilities the benchmarks are intended to measure, and call this benchmark maintenance debt.","excerpt":"We call the resulting accumulation of defects benchmark maintenance debt: a growing gap between benchmark scores and the capabilities they are intended to measure.","support":"source-stated"},{"text":"The authors report that their audit of Terminal-Bench 2.0 found issues in roughly 30% of the benchmark, and that the repairs changed evaluation results.","excerpt":"Our audit identifies issues in roughly 30% of the benchmark, and we release the resulting repairs as Terminal-Bench 2.1.","support":"source-stated"},{"text":"The authors report that repaired evaluation results increased accuracy by over 10 percentage points and moved leaderboard positions by up to three ranks, while overall rankings remained highly correlated.","excerpt":"accuracy increased by over 10 percentage points and leaderboard positions move by up to three ranks, even as the overall ranking remains highly correlated (Spearman rho = 0.958).","support":"source-stated"},{"text":"The authors argue that resource budgets, including fixed wall-clock limits, are part of the measurement specification and should be validated with benchmark tasks.","excerpt":"resource budgets are part of the measurement specification and must be validated alongside the tasks themselves.","support":"source-stated"},{"text":"The authors report that extending the time limit closed a 20pp harness gap on GPT 5.4 and reduced a 30 pp gap on GPT 5.4 mini to 5 pp.","excerpt":"extending the time limit closes a 20pp harness gap on GPT 5.4 and shrinks a 30 pp gap on GPT 5.4 mini to 5 pp.","support":"source-stated"}],"limitations":["The evidence is the arXiv abstract; it does not provide the audit methods, task-level findings or details needed to assess how the reported results were obtained.","The abstract reports results for Terminal-Bench 2.0 and does not establish how well the workflow generalizes to other benchmarks."],"mediaId":"f829c1d2e9578a614af16090fe4c66ea89bab608f7757c3f9ad1d0bce7e7d4a5","correctionReason":null,"id":"a3a4f4a3d8deceaffeb461bb2d13a47a88ead05f3b612b41805e290fc581ebbd","publicationId":"6de9593e-f0d2-411f-8fd6-ea9800209598","version":1,"sourceUrl":"https://arxiv.org/abs/2611.08417","sourceTitle":"Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-05","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:10:38.570Z","createdAt":"2026-11-08T22:11:11.953Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"f829c1d2e9578a614af16090fe4c66ea89bab608f7757c3f9ad1d0bce7e7d4a5","kind":"article","url":"https://arxiv.org/abs/2611.08417","sourceUrl":"https://arxiv.org/abs/2611.08417","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-08T22:10:06.611Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.08417","checkedAt":"2026-11-08T22:42:09.312Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"f829c1d2e9578a614af16090fe4c66ea89bab608f7757c3f9ad1d0bce7e7d4a5","confidence":0.9400000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"c2c72b1aed3fa5636537d559d1ba7e95ce25ed7dbc29ea59c3a6ed0a4b7515e1","decidedAt":"2026-11-08T22:10:10.907Z"},"provenance":{"textHash":"754af07b4e64cf4fb18d52b0519bb5fa99d663a65b50c7852304663f89573951","observationId":"b221a7ac6e9eae299c52c5dd24954461ecd228bbc3095b7f767ed6de909bb208","retrievedAt":"2026-11-08T22:10:06.083Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"fcd1adca-6bf3-4d29-85ea-039aa166ccd9"},"reviewer":{"model":"gpt-6-luna","runId":"ef82e9e0-907a-473f-8748-bf97fb090c23"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-08T22:11:11.953Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Buchanan, E. Kelly","url":null,"kind":"unknown"},{"name":"Zhou, Crystal","url":null,"kind":"unknown"},{"name":"Linderman, Scott W.","url":null,"kind":"unknown"},{"name":"Ré, Christopher","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Buchanan, E. Kelly","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Buchanan, E. Kelly","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.08417","locator":"head/meta[name=author|citation_author]"},"assertionId":"0ab3f7a7783eda80a0df403a50fb0481f2d8fefc57e5b7a08da76fe5b3fff58b:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T22:42:09.312Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"0ab3f7a7783eda80a0df403a50fb0481f2d8fefc57e5b7a08da76fe5b3fff58b","sourceUrl":"https://arxiv.org/abs/2611.08417","contentSha256":"a594919dd0565a9653f70e80ad0649871d8eb6ae05c0cb71d8e6bcca3b667609","textHash":"754af07b4e64cf4fb18d52b0519bb5fa99d663a65b50c7852304663f89573951","observedAt":"2026-11-08T22:42:09.312Z"}},{"actor":{"name":"Zhou, Crystal","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhou, Crystal","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.08417","locator":"head/meta[name=author|citation_author]"},"assertionId":"0ab3f7a7783eda80a0df403a50fb0481f2d8fefc57e5b7a08da76fe5b3fff58b:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T22:42:09.312Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"0ab3f7a7783eda80a0df403a50fb0481f2d8fefc57e5b7a08da76fe5b3fff58b","sourceUrl":"https://arxiv.org/abs/2611.08417","contentSha256":"a594919dd0565a9653f70e80ad0649871d8eb6ae05c0cb71d8e6bcca3b667609","textHash":"754af07b4e64cf4fb18d52b0519bb5fa99d663a65b50c7852304663f89573951","observedAt":"2026-11-08T22:42:09.312Z"}},{"actor":{"name":"Linderman, Scott W.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Linderman, Scott W.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.08417","locator":"head/meta[name=author|citation_author]"},"assertionId":"0ab3f7a7783eda80a0df403a50fb0481f2d8fefc57e5b7a08da76fe5b3fff58b:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T22:42:09.312Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"0ab3f7a7783eda80a0df403a50fb0481f2d8fefc57e5b7a08da76fe5b3fff58b","sourceUrl":"https://arxiv.org/abs/2611.08417","contentSha256":"a594919dd0565a9653f70e80ad0649871d8eb6ae05c0cb71d8e6bcca3b667609","textHash":"754af07b4e64cf4fb18d52b0519bb5fa99d663a65b50c7852304663f89573951","observedAt":"2026-11-08T22:42:09.312Z"}},{"actor":{"name":"Ré, Christopher","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ré, Christopher","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.08417","locator":"head/meta[name=author|citation_author]"},"assertionId":"0ab3f7a7783eda80a0df403a50fb0481f2d8fefc57e5b7a08da76fe5b3fff58b:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T22:42:09.312Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"0ab3f7a7783eda80a0df403a50fb0481f2d8fefc57e5b7a08da76fe5b3fff58b","sourceUrl":"https://arxiv.org/abs/2611.08417","contentSha256":"a594919dd0565a9653f70e80ad0649871d8eb6ae05c0cb71d8e6bcca3b667609","textHash":"754af07b4e64cf4fb18d52b0519bb5fa99d663a65b50c7852304663f89573951","observedAt":"2026-11-08T22:42:09.312Z"}}],"participationCoverage":{"total":4,"returned":4,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.08417","snapshotId":"0ab3f7a7783eda80a0df403a50fb0481f2d8fefc57e5b7a08da76fe5b3fff58b","contentSha256":"a594919dd0565a9653f70e80ad0649871d8eb6ae05c0cb71d8e6bcca3b667609","textHash":"754af07b4e64cf4fb18d52b0519bb5fa99d663a65b50c7852304663f89573951","observedAt":"2026-11-08T22:42:09.312Z","basis":"publisher-page-metadata","sourceLinks":42,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/a3a4f4a3d8deceaffeb461bb2d13a47a88ead05f3b612b41805e290fc581ebbd/source"}}}],"audits":[{"publicationId":"6de9593e-f0d2-411f-8fd6-ea9800209598","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading correctly names the four listed authors and the arXiv cs.AI venue. It frames the findings as the authors' arguments and reported results rather than as established facts.","excerpt":"Authors:E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman, Christopher Ré"},{"dimension":"summary","status":"uncertain","explanation":"The reading is faithful, but central terms are left unexplained for non-specialists: Terminal-Bench (a coding-agent benchmark), verifiers, wall-clock, harness gap and pp. It also drops the stakes the source gives: coding agents' economic consequence and that Terminal-Bench 2.0 had extensive review and red-teaming before release, which gives the 30% figure its force. Repair: add one plain-language framing sentence and tooltips. The question also presupposes the workflow already validates resource budgets; the abstract only says budgets must be validated.","excerpt":"Coding agents are among the most economically consequential deployments of LLMs to date, making reliable evaluation increasingly important."},{"dimension":"date","status":"pass","explanation":"The displayed original publication date, 2026-11-05, matches the arXiv v1 submission date. It is not the later feed date.","excerpt":"[Submitted on 5 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"Every listed claim and figure is directly stated in the abstract: the 30% of tasks with issues, the more-than-10-point accuracy increase, rank moves of up to three places, high ranking correlation, and the 20pp and 30-to-5pp gap results. The limitations correctly note that this is abstract-only evidence. No benefits or costs are invented.","excerpt":"Our audit identifies issues in roughly 30% of the benchmark, and we release the resulting repairs as Terminal-Bench 2.1."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.","excerpt":null}],"createdAt":"2026-11-08T22:11:11.953Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}