Hua, Yining · Nov 17, 2026
arXiv cs.AI paper: Higher-Scoring Coding Models Did Not Get More Work Done
Markdown · JSON · Original source
In the abstract, author Yining Hua argues that higher coding- scores did not translate into significantly more completed work in the comparisons studied. The paper treats a 19-day suspension of access to Fable 5 as a natural experiment: users switched models while their people and projects stayed the same. It compares GitHub pull requests from 297 Fable 5 users before, during and after the suspension with those of 8,908 developers who remained on Opus 4.8. The abstract reports that access to Fable 5 was associated with a 1.4-percentage-point increase in the 7-day merge rate, but this was not . Across five model upgrades, the weighted mean change was negative and also not statistically significant; the abstract likewise reports no significant increase in pull requests merged per week. Escape Meridian interpretation: these results challenge the assumption that better benchmark scores mean more work gets done, but the evidence concerns these model changes and GitHub measures, not every form of coding work.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 17, 2026 (2026-11-17).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-18T18:33:29.138Z. First observed: 2026-11-18T05:07:08.541Z. Retrieved: 2026-11-18T18:32:43.594Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The evidence supplied is the abstract, not the full paper; details of the study methods and analyses beyond the abstract are unavailable here. The reported outcomes are GitHub pull-request merge rates and numbers of merged pull requests; the abstract does not establish how the results apply to other measures of coding work.
Open question
Would the reported pattern hold for measures of coding work beyond GitHub pull requests?
Source-stated
Hua frames the study as a test of whether higher-scoring coding models lead to more real work getting done.
no study has tested if a model that scores higher on these benchmarks gets more real work done.
Source-stated
The study compares GitHub pull requests during a 19-day suspension of access to Claude Fable 5 with pull requests from developers who stayed on Opus 4.8.
We tracked GitHub pull requests of 297 Fable 5 users before, during, and after the suspension. We used 8,908 developers who stayed on Opus 4.8 all the time as the control group.
Source-stated
The abstract reports a 1.4-percentage-point increase in the 7-day merge rate associated with access to Fable 5, which was not statistically significant.
increased the 7-day merge rate of pull requests by only 1.4 percentage points [-1.4; +4.3], which was not statistically significant.
Source-stated
The abstract reports no significant increase in pull requests merged per week.
We also did not observe a significant increase in the number of pull requests merged per week (+2.4% [-4.9%; +10.2%]).
Escape Meridian interpretation
The reported results concern GitHub pull-request measures in the described suspension and model-upgrade comparisons; whether they extend to other measures of coding work remains open.
We tracked GitHub pull requests of 297 Fable 5 users before, during, and after the suspension.
Source and contributor context
Hua, Yining · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Statistical significance
A test of whether a result is unlikely to be due to chance alone; it does not say how large or important an effect is.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: 4a975edd-5113-40de-8d9f-4956f09d2354.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T18:33:29.138Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 55be3cc82753e8b247ead19561800ca49401c96f1f1edf0e577e773f955c2a6c. Source text hash: 9b3236e2b14c6a0a2785a826354fbbabae559c60aae6474174796c9fd6fcb741.
Revision and source-check history
Version 1 · 2026-11-18T18:33:29.138Z
arXiv cs.AI paper: Higher-Scoring Coding Models Did Not Get More Work Done
In the abstract, author Yining Hua argues that higher coding- scores did not translate into significantly more completed work in the comparisons studied. The paper treats a 19-day suspension of access to Fable 5 as a natural experiment: users switched models while their people and projects stayed the same. It compares GitHub pull requests from 297 Fable 5 users before, during and after the suspension with those of 8,908 developers who remained on Opus 4.8. The abstract reports that access to Fable 5 was associated with a 1.4-percentage-point increase in the 7-day merge rate, but this was not . Across five model upgrades, the weighted mean change was negative and also not statistically significant; the abstract likewise reports no significant increase in pull requests merged per week. Escape Meridian interpretation: these results challenge the assumption that better benchmark scores mean more work gets done, but the evidence concerns these model changes and GitHub measures, not every form of coding work.
Original source
cs.AI · Article
Source details
Original source date: Nov 17, 2026 (2026-11-17).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-18T18:33:29.138Z. First observed: 2026-11-18T05:07:08.541Z. Retrieved: 2026-11-18T18:32:43.594Z.
Limits
The evidence supplied is the abstract, not the full paper; details of the study methods and analyses beyond the abstract are unavailable here. The reported outcomes are GitHub pull-request merge rates and numbers of merged pull requests; the abstract does not establish how the results apply to other measures of coding work.
Open question
Would the reported pattern hold for measures of coding work beyond GitHub pull requests?
Source-stated
Hua frames the study as a test of whether higher-scoring coding models lead to more real work getting done.
no study has tested if a model that scores higher on these benchmarks gets more real work done.
Source-stated
The study compares GitHub pull requests during a 19-day suspension of access to Claude Fable 5 with pull requests from developers who stayed on Opus 4.8.
We tracked GitHub pull requests of 297 Fable 5 users before, during, and after the suspension. We used 8,908 developers who stayed on Opus 4.8 all the time as the control group.
Source-stated
The abstract reports a 1.4-percentage-point increase in the 7-day merge rate associated with access to Fable 5, which was not statistically significant.
increased the 7-day merge rate of pull requests by only 1.4 percentage points [-1.4; +4.3], which was not statistically significant.
Source-stated
The abstract reports no significant increase in pull requests merged per week.
We also did not observe a significant increase in the number of pull requests merged per week (+2.4% [-4.9%; +10.2%]).
Escape Meridian interpretation
The reported results concern GitHub pull-request measures in the described suspension and model-upgrade comparisons; whether they extend to other measures of coding work remains open.
We tracked GitHub pull requests of 297 Fable 5 users before, during, and after the suspension.
Source and contributor context
Hua, Yining · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Statistical significance
A test of whether a result is unlikely to be due to chance alone; it does not say how large or important an effect is.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: 4a975edd-5113-40de-8d9f-4956f09d2354.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T18:33:29.138Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 55be3cc82753e8b247ead19561800ca49401c96f1f1edf0e577e773f955c2a6c. Source text hash: 9b3236e2b14c6a0a2785a826354fbbabae559c60aae6474174796c9fd6fcb741.
uncertain · 2026-11-18T18:33:29.138Z · anthropic/claude-opus-5.5
Publication: 4a975edd-5113-40de-8d9f-4956f09d2354
attribution — pass: The reading attributes the study and its framing to author Yining Hua in an arXiv abstract. The paper's primary listing is cs.SE with a cs.AI cross-list, so the cs.AI label is acceptable. Results are presented as what the abstract reports.
Authors:Yining Hua
summary — uncertain: The summary leaves out why the comparison tests benchmarks at all: most users switched to Opus 4.8, which the source says scores 11 points lower on SWE-Bench Pro. It also omits the cause of the suspension, a U.S. export control directive. Smallest repair: add one sentence covering both points. Separately, the central measures, pull requests and 7-day merge rate, are not explained in plain language. Add tooltips or a brief gloss. The omitted detail that all five upgrade estimates were negative would also preserve the source's force.
most former Fable 5 users switched to Opus 4.8 (11 points lower on SWE-Bench Pro compared to Fable 5)
date — pass: originalPublished 2026-11-17 matches the v1 submission date shown on the arXiv page.
[Submitted on 17 Nov 2026]
support — pass: The numeric claims, the non-significance and the study design match the abstract. Saying the effect was 'associated with' access is more cautious than the source's 'increased', not an overstatement. The Escape Meridian interpretation is labeled and scoped to these model changes and GitHub measures. The question follows from the evidence without leading.
no significant increases of merge rates were found with the upgrades (weighted mean: -0.9 percentage points [-1.9; +0.2])
media — pass: The selected media is the verified direct link to the original arXiv abstract page, which is the source itself.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeA test of whether a result is unlikely to be due to chance alone; it does not say how large or important an effect is.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeA test of whether a result is unlikely to be due to chance alone; it does not say how large or important an effect is.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv