Buchanan, E. Kelly, Zhou, Crystal, Linderman, Scott W., Ré, Christopher · Nov 5, 2026
arXiv cs.AI: Buchanan et al. argue benchmarks need continuous maintenance
Markdown · JSON · Original source
E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman and Christopher Ré argue that can become stale even after careful validation. As models improve, they may find valid solutions that verifiers reject; changes to evaluation infrastructure can also undermine assumptions built into tasks and tests. The authors call the resulting gap between scores and intended capabilities “benchmark maintenance debt.” They describe a continuous-validation workflow combining community reports, task artifacts, agent traces, automated oracle checks and human-gated repairs. Applying it to Terminal-Bench 2.0, they report issues in roughly 30% of the benchmark and release repairs as Terminal-Bench 2.1. The authors say those repairs raised accuracy by over 10 percentage points and moved leaderboard positions by as many as three ranks, while overall rankings remained highly correlated. Their argument goes beyond fixing tasks: fixed wall-clock limits can change agent performance, so resource budgets are part of what a benchmark measures and need validation too. They report that extending the limit closed a 20pp harness gap on GPT 5.4. For GPT 5.4 mini, it reduced a 30 pp gap to 5 pp. The abstract describes these results but does not detail the audit methods or how the workflow generalizes beyond this benchmark.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 5, 2026 (2026-11-05).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-08T22:11:11.953Z. First observed: 2026-11-06T05:10:38.570Z. Retrieved: 2026-11-08T22:10:06.083Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The evidence is the abstract; it does not provide the audit methods, task-level findings or details needed to assess how the reported results were obtained. The abstract reports results for Terminal-Bench 2.0 and does not establish how well the workflow generalizes to other .
Open question
How does the authors’ workflow validate resource budgets alongside task artifacts as models and evaluation infrastructure change?
Source-stated
The authors argue that benchmark defects can cause scores to diverge from the capabilities the benchmarks are intended to measure, and call this benchmark maintenance debt.
We call the resulting accumulation of defects benchmark maintenance debt: a growing gap between benchmark scores and the capabilities they are intended to measure.
Source-stated
The authors report that their audit of Terminal-Bench 2.0 found issues in roughly 30% of the benchmark, and that the repairs changed evaluation results.
Our audit identifies issues in roughly 30% of the benchmark, and we release the resulting repairs as Terminal-Bench 2.1.
Source-stated
The authors report that repaired evaluation results increased accuracy by over 10 percentage points and moved leaderboard positions by up to three ranks, while overall rankings remained highly correlated.
accuracy increased by over 10 percentage points and leaderboard positions move by up to three ranks, even as the overall ranking remains highly correlated (Spearman rho = 0.958).
Source-stated
The authors argue that resource budgets, including fixed wall-clock limits, are part of the measurement specification and should be validated with benchmark tasks.
resource budgets are part of the measurement specification and must be validated alongside the tasks themselves.
Source-stated
The authors report that extending the time limit closed a 20pp harness gap on GPT 5.4 and reduced a 30 pp gap on GPT 5.4 mini to 5 pp.
extending the time limit closes a 20pp harness gap on GPT 5.4 and shrinks a 30 pp gap on GPT 5.4 mini to 5 pp.
Source and contributor context
Buchanan, E. Kelly, Zhou, Crystal, Linderman, Scott W., Ré, Christopher · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 6de9593e-f0d2-411f-8fd6-ea9800209598.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-08T22:11:11.953Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: b221a7ac6e9eae299c52c5dd24954461ecd228bbc3095b7f767ed6de909bb208. Source text hash: 754af07b4e64cf4fb18d52b0519bb5fa99d663a65b50c7852304663f89573951.
Revision and source-check history
Version 1 · 2026-11-08T22:11:11.953Z
arXiv cs.AI: Buchanan et al. argue benchmarks need continuous maintenance
E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman and Christopher Ré argue that can become stale even after careful validation. As models improve, they may find valid solutions that verifiers reject; changes to evaluation infrastructure can also undermine assumptions built into tasks and tests. The authors call the resulting gap between scores and intended capabilities “benchmark maintenance debt.” They describe a continuous-validation workflow combining community reports, task artifacts, agent traces, automated oracle checks and human-gated repairs. Applying it to Terminal-Bench 2.0, they report issues in roughly 30% of the benchmark and release repairs as Terminal-Bench 2.1. The authors say those repairs raised accuracy by over 10 percentage points and moved leaderboard positions by as many as three ranks, while overall rankings remained highly correlated. Their argument goes beyond fixing tasks: fixed wall-clock limits can change agent performance, so resource budgets are part of what a benchmark measures and need validation too. They report that extending the limit closed a 20pp harness gap on GPT 5.4. For GPT 5.4 mini, it reduced a 30 pp gap to 5 pp. The abstract describes these results but does not detail the audit methods or how the workflow generalizes beyond this benchmark.
Original source
cs.AI · Article
Source details
Original source date: Nov 5, 2026 (2026-11-05).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-08T22:11:11.953Z. First observed: 2026-11-06T05:10:38.570Z. Retrieved: 2026-11-08T22:10:06.083Z.
Limits
The evidence is the abstract; it does not provide the audit methods, task-level findings or details needed to assess how the reported results were obtained. The abstract reports results for Terminal-Bench 2.0 and does not establish how well the workflow generalizes to other .
Open question
How does the authors’ workflow validate resource budgets alongside task artifacts as models and evaluation infrastructure change?
Source-stated
The authors argue that benchmark defects can cause scores to diverge from the capabilities the benchmarks are intended to measure, and call this benchmark maintenance debt.
We call the resulting accumulation of defects benchmark maintenance debt: a growing gap between benchmark scores and the capabilities they are intended to measure.
Source-stated
The authors report that their audit of Terminal-Bench 2.0 found issues in roughly 30% of the benchmark, and that the repairs changed evaluation results.
Our audit identifies issues in roughly 30% of the benchmark, and we release the resulting repairs as Terminal-Bench 2.1.
Source-stated
The authors report that repaired evaluation results increased accuracy by over 10 percentage points and moved leaderboard positions by up to three ranks, while overall rankings remained highly correlated.
accuracy increased by over 10 percentage points and leaderboard positions move by up to three ranks, even as the overall ranking remains highly correlated (Spearman rho = 0.958).
Source-stated
The authors argue that resource budgets, including fixed wall-clock limits, are part of the measurement specification and should be validated with benchmark tasks.
resource budgets are part of the measurement specification and must be validated alongside the tasks themselves.
Source-stated
The authors report that extending the time limit closed a 20pp harness gap on GPT 5.4 and reduced a 30 pp gap on GPT 5.4 mini to 5 pp.
extending the time limit closes a 20pp harness gap on GPT 5.4 and shrinks a 30 pp gap on GPT 5.4 mini to 5 pp.
Source and contributor context
Buchanan, E. Kelly, Zhou, Crystal, Linderman, Scott W., Ré, Christopher · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 6de9593e-f0d2-411f-8fd6-ea9800209598.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-08T22:11:11.953Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: b221a7ac6e9eae299c52c5dd24954461ecd228bbc3095b7f767ed6de909bb208. Source text hash: 754af07b4e64cf4fb18d52b0519bb5fa99d663a65b50c7852304663f89573951.
uncertain · 2026-11-08T22:11:11.953Z · anthropic/claude-opus-5.5
Publication: 6de9593e-f0d2-411f-8fd6-ea9800209598
attribution — pass: The reading correctly names the four listed authors and the arXiv cs.AI venue. It frames the findings as the authors' arguments and reported results rather than as established facts.
Authors:E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman, Christopher Ré
summary — uncertain: The reading is faithful, but central terms are left unexplained for non-specialists: Terminal-Bench (a coding-agent benchmark), verifiers, wall-clock, harness gap and pp. It also drops the stakes the source gives: coding agents' economic consequence and that Terminal-Bench 2.0 had extensive review and red-teaming before release, which gives the 30% figure its force. Repair: add one plain-language framing sentence and tooltips. The question also presupposes the workflow already validates resource budgets; the abstract only says budgets must be validated.
Coding agents are among the most economically consequential deployments of LLMs to date, making reliable evaluation increasingly important.
date — pass: The displayed original publication date, 2026-11-05, matches the arXiv v1 submission date. It is not the later feed date.
[Submitted on 5 Nov 2026]
support — pass: Every listed claim and figure is directly stated in the abstract: the 30% of tasks with issues, the more-than-10-point accuracy increase, rank moves of up to three places, high ranking correlation, and the 20pp and 30-to-5pp gap results. The limitations correctly note that this is abstract-only evidence. No benefits or costs are invented.
Our audit identifies issues in roughly 30% of the benchmark, and we release the resulting repairs as Terminal-Bench 2.1.
media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference