{"id":"983a6bbd5cbde3fc6ba856e75dbbef08b5700b63639907064d23664b058a9e18","title":"arXiv cs.AI paper: Higher-Scoring Coding Models Did Not Get More Work Done","argument":"In the arXiv abstract, author Yining Hua argues that higher coding-benchmark scores did not translate into significantly more completed work in the comparisons studied. The paper treats a 19-day suspension of access to Claude Fable 5 as a natural experiment: users switched models while their people and projects stayed the same. It compares GitHub pull requests from 297 Fable 5 users before, during and after the suspension with those of 8,908 developers who remained on Opus 4.8. The abstract reports that access to Fable 5 was associated with a 1.4-percentage-point increase in the 7-day merge rate, but this was not statistically significant. Across five model upgrades, the weighted mean change was negative and also not statistically significant; the abstract likewise reports no significant increase in pull requests merged per week. Escape Meridian interpretation: these results challenge the assumption that better benchmark scores mean more work gets done, but the evidence concerns these model changes and GitHub measures, not every form of coding work.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence supplied is the arXiv abstract, not the full paper; details of the study methods and analyses beyond the abstract are unavailable here. The reported outcomes are GitHub pull-request merge rates and numbers of merged pull requests; the abstract does not establish how the results apply to other measures of coding work.","speaker":{"name":"Hua, Yining","roleAtSource":"Article author"},"source":{"title":"Higher-Scoring Coding Models Did Not Get More Work Done","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.23216","publishedAt":"2026-11-17","dateBasis":"Date basis not provided."},"evidence":[{"text":"Hua frames the study as a test of whether higher-scoring coding models lead to more real work getting done.","excerpt":"no study has tested if a model that scores higher on these benchmarks gets more real work done.","url":"https://arxiv.org/abs/2611.23216","relationship":"Source-stated"},{"text":"The study compares GitHub pull requests during a 19-day suspension of access to Claude Fable 5 with pull requests from developers who stayed on Opus 4.8.","excerpt":"We tracked GitHub pull requests of 297 Fable 5 users before, during, and after the suspension. We used 8,908 developers who stayed on Opus 4.8 all the time as the control group.","url":"https://arxiv.org/abs/2611.23216","relationship":"Source-stated"},{"text":"The abstract reports a 1.4-percentage-point increase in the 7-day merge rate associated with access to Fable 5, which was not statistically significant.","excerpt":"increased the 7-day merge rate of pull requests by only 1.4 percentage points [-1.4; +4.3], which was not statistically significant.","url":"https://arxiv.org/abs/2611.23216","relationship":"Source-stated"},{"text":"The abstract reports no significant increase in pull requests merged per week.","excerpt":"We also did not observe a significant increase in the number of pull requests merged per week (+2.4% [-4.9%; +10.2%]).","url":"https://arxiv.org/abs/2611.23216","relationship":"Source-stated"},{"text":"The reported results concern GitHub pull-request measures in the described suspension and model-upgrade comparisons; whether they extend to other measures of coding work remains open.","excerpt":"We tracked GitHub pull requests of 297 Fable 5 users before, during, and after the suspension.","url":"https://arxiv.org/abs/2611.23216","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.23216","checkedAt":"2026-11-18T18:48:11.778Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Would the reported pattern hold for measures of coding work beyond GitHub pull requests?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI paper: Higher-Scoring Coding Models Did Not Get More Work Done","summary":"In the arXiv abstract, author Yining Hua argues that higher coding-benchmark scores did not translate into significantly more completed work in the comparisons studied. The paper treats a 19-day suspension of access to Claude Fable 5 as a natural experiment: users switched models while their people and projects stayed the same. It compares GitHub pull requests from 297 Fable 5 users before, during and after the suspension with those of 8,908 developers who remained on Opus 4.8. The abstract reports that access to Fable 5 was associated with a 1.4-percentage-point increase in the 7-day merge rate, but this was not statistically significant. Across five model upgrades, the weighted mean change was negative and also not statistically significant; the abstract likewise reports no significant increase in pull requests merged per week. Escape Meridian interpretation: these results challenge the assumption that better benchmark scores mean more work gets done, but the evidence concerns these model changes and GitHub measures, not every form of coding work.","question":"Would the reported pattern hold for measures of coding work beyond GitHub pull requests?","claims":[{"text":"Hua frames the study as a test of whether higher-scoring coding models lead to more real work getting done.","excerpt":"no study has tested if a model that scores higher on these benchmarks gets more real work done.","support":"source-stated"},{"text":"The study compares GitHub pull requests during a 19-day suspension of access to Claude Fable 5 with pull requests from developers who stayed on Opus 4.8.","excerpt":"We tracked GitHub pull requests of 297 Fable 5 users before, during, and after the suspension. We used 8,908 developers who stayed on Opus 4.8 all the time as the control group.","support":"source-stated"},{"text":"The abstract reports a 1.4-percentage-point increase in the 7-day merge rate associated with access to Fable 5, which was not statistically significant.","excerpt":"increased the 7-day merge rate of pull requests by only 1.4 percentage points [-1.4; +4.3], which was not statistically significant.","support":"source-stated"},{"text":"The abstract reports no significant increase in pull requests merged per week.","excerpt":"We also did not observe a significant increase in the number of pull requests merged per week (+2.4% [-4.9%; +10.2%]).","support":"source-stated"},{"text":"The reported results concern GitHub pull-request measures in the described suspension and model-upgrade comparisons; whether they extend to other measures of coding work remains open.","excerpt":"We tracked GitHub pull requests of 297 Fable 5 users before, during, and after the suspension.","support":"interpretation"}],"limitations":["The evidence supplied is the arXiv abstract, not the full paper; details of the study methods and analyses beyond the abstract are unavailable here.","The reported outcomes are GitHub pull-request merge rates and numbers of merged pull requests; the abstract does not establish how the results apply to other measures of coding work."],"mediaId":"b49f06a805c7d88e1513568277b2f225c91f5eaba0bb1e2600e73f58eaefb0b3","correctionReason":null,"id":"983a6bbd5cbde3fc6ba856e75dbbef08b5700b63639907064d23664b058a9e18","publicationId":"4a975edd-5113-40de-8d9f-4956f09d2354","version":1,"sourceUrl":"https://arxiv.org/abs/2611.23216","sourceTitle":"Higher-Scoring Coding Models Did Not Get More Work Done","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-17","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-18T05:07:08.541Z","createdAt":"2026-11-18T18:33:29.138Z","historical":false,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"b49f06a805c7d88e1513568277b2f225c91f5eaba0bb1e2600e73f58eaefb0b3","kind":"article","url":"https://arxiv.org/abs/2611.23216","sourceUrl":"https://arxiv.org/abs/2611.23216","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-18T18:32:43.769Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.23216","checkedAt":"2026-11-18T18:48:11.778Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"b49f06a805c7d88e1513568277b2f225c91f5eaba0bb1e2600e73f58eaefb0b3","confidence":0.98,"provider":"typesafe","model":"jev-1.13.0","inputHash":"7554b548f069f0a874f56cf1246dc84135d9fef78f678c18208eefe894e9d9f9","decidedAt":"2026-11-18T18:32:44.995Z"},"provenance":{"textHash":"9b3236e2b14c6a0a2785a826354fbbabae559c60aae6474174796c9fd6fcb741","observationId":"55be3cc82753e8b247ead19561800ca49401c96f1f1edf0e577e773f955c2a6c","retrievedAt":"2026-11-18T18:32:43.594Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"7580f6fc-efba-4a38-8259-741a6c13f13a"},"reviewer":{"model":"gpt-6-luna","runId":"0c60be9a-d55f-4a7c-8bed-a221cbe12b1a"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-18T18:33:29.138Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Hua, Yining","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Hua, Yining","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Hua, Yining","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.23216","locator":"head/meta[name=author|citation_author]"},"assertionId":"c9725c53c15665ff21cb57d413cfaa270b0fb983cbb9392847bcdbb488a39d8e:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T18:48:11.778Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c9725c53c15665ff21cb57d413cfaa270b0fb983cbb9392847bcdbb488a39d8e","sourceUrl":"https://arxiv.org/abs/2611.23216","contentSha256":"34f4b00979c01edcbfb0661f345f053fff1504b40ea889e55a7698c20b115b1c","textHash":"9b3236e2b14c6a0a2785a826354fbbabae559c60aae6474174796c9fd6fcb741","observedAt":"2026-11-18T18:48:11.778Z"}}],"participationCoverage":{"total":1,"returned":1,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.23216","snapshotId":"c9725c53c15665ff21cb57d413cfaa270b0fb983cbb9392847bcdbb488a39d8e","contentSha256":"34f4b00979c01edcbfb0661f345f053fff1504b40ea889e55a7698c20b115b1c","textHash":"9b3236e2b14c6a0a2785a826354fbbabae559c60aae6474174796c9fd6fcb741","observedAt":"2026-11-18T18:48:11.778Z","basis":"publisher-page-metadata","sourceLinks":41,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/983a6bbd5cbde3fc6ba856e75dbbef08b5700b63639907064d23664b058a9e18/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-paper-higher-scoring-coding-models-did-not-get-more-983a6bbd","formats":{"html":"/readings/arxiv-cs-ai-paper-higher-scoring-coding-models-did-not-get-more-983a6bbd","markdown":"/readings/arxiv-cs-ai-paper-higher-scoring-coding-models-did-not-get-more-983a6bbd.md","json":"/readings/arxiv-cs-ai-paper-higher-scoring-coding-models-did-not-get-more-983a6bbd.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-statistical-significance","name":"Statistical significance","explanation":"A test of whether a result is unlikely to be due to chance alone; it does not say how large or important an effect is.","referenceUrl":"https://en.wikipedia.org/wiki/Statistical_significance"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"},{"id":"claude","name":"Claude","kind":"model","context":"Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.","referenceUrls":["https://claude.com/product/overview","https://www.anthropic.com/news","https://www.anthropic.com/research/team/economics"],"path":"/entities/claude","json":"/entities/claude.json","markdown":"/entities/claude.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"983a6bbd5cbde3fc6ba856e75dbbef08b5700b63639907064d23664b058a9e18","revision":"4a975edd-5113-40de-8d9f-4956f09d2354","revisionKind":"publication-id","title":"arXiv cs.AI paper: Higher-Scoring Coding Models Did Not Get More Work Done","url":"https://escapemeridian.com/readings/arxiv-cs-ai-paper-higher-scoring-coding-models-did-not-get-more-983a6bbd"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.23216","revision":"9b3236e2b14c6a0a2785a826354fbbabae559c60aae6474174796c9fd6fcb741","revisionKind":"sha256-source-text","title":"Higher-Scoring Coding Models Did Not Get More Work Done","url":"https://arxiv.org/abs/2611.23216"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"983a6bbd5cbde3fc6ba856e75dbbef08b5700b63639907064d23664b058a9e18","title":"arXiv cs.AI paper: Higher-Scoring Coding Models Did Not Get More Work Done","argument":"In the arXiv abstract, author Yining Hua argues that higher coding-benchmark scores did not translate into significantly more completed work in the comparisons studied. The paper treats a 19-day suspension of access to Claude Fable 5 as a natural experiment: users switched models while their people and projects stayed the same. It compares GitHub pull requests from 297 Fable 5 users before, during and after the suspension with those of 8,908 developers who remained on Opus 4.8. The abstract reports that access to Fable 5 was associated with a 1.4-percentage-point increase in the 7-day merge rate, but this was not statistically significant. Across five model upgrades, the weighted mean change was negative and also not statistically significant; the abstract likewise reports no significant increase in pull requests merged per week. Escape Meridian interpretation: these results challenge the assumption that better benchmark scores mean more work gets done, but the evidence concerns these model changes and GitHub measures, not every form of coding work.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence supplied is the arXiv abstract, not the full paper; details of the study methods and analyses beyond the abstract are unavailable here. The reported outcomes are GitHub pull-request merge rates and numbers of merged pull requests; the abstract does not establish how the results apply to other measures of coding work.","speaker":{"name":"Hua, Yining","roleAtSource":"Article author"},"source":{"title":"Higher-Scoring Coding Models Did Not Get More Work Done","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.23216","publishedAt":"2026-11-17","dateBasis":"Date basis not provided."},"evidence":[{"text":"Hua frames the study as a test of whether higher-scoring coding models lead to more real work getting done.","excerpt":"no study has tested if a model that scores higher on these benchmarks gets more real work done.","url":"https://arxiv.org/abs/2611.23216","relationship":"Source-stated"},{"text":"The study compares GitHub pull requests during a 19-day suspension of access to Claude Fable 5 with pull requests from developers who stayed on Opus 4.8.","excerpt":"We tracked GitHub pull requests of 297 Fable 5 users before, during, and after the suspension. We used 8,908 developers who stayed on Opus 4.8 all the time as the control group.","url":"https://arxiv.org/abs/2611.23216","relationship":"Source-stated"},{"text":"The abstract reports a 1.4-percentage-point increase in the 7-day merge rate associated with access to Fable 5, which was not statistically significant.","excerpt":"increased the 7-day merge rate of pull requests by only 1.4 percentage points [-1.4; +4.3], which was not statistically significant.","url":"https://arxiv.org/abs/2611.23216","relationship":"Source-stated"},{"text":"The abstract reports no significant increase in pull requests merged per week.","excerpt":"We also did not observe a significant increase in the number of pull requests merged per week (+2.4% [-4.9%; +10.2%]).","url":"https://arxiv.org/abs/2611.23216","relationship":"Source-stated"},{"text":"The reported results concern GitHub pull-request measures in the described suspension and model-upgrade comparisons; whether they extend to other measures of coding work remains open.","excerpt":"We tracked GitHub pull requests of 297 Fable 5 users before, during, and after the suspension.","url":"https://arxiv.org/abs/2611.23216","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.23216","checkedAt":"2026-11-18T18:48:11.778Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Would the reported pattern hold for measures of coding work beyond GitHub pull requests?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI paper: Higher-Scoring Coding Models Did Not Get More Work Done","summary":"In the arXiv abstract, author Yining Hua argues that higher coding-benchmark scores did not translate into significantly more completed work in the comparisons studied. The paper treats a 19-day suspension of access to Claude Fable 5 as a natural experiment: users switched models while their people and projects stayed the same. It compares GitHub pull requests from 297 Fable 5 users before, during and after the suspension with those of 8,908 developers who remained on Opus 4.8. The abstract reports that access to Fable 5 was associated with a 1.4-percentage-point increase in the 7-day merge rate, but this was not statistically significant. Across five model upgrades, the weighted mean change was negative and also not statistically significant; the abstract likewise reports no significant increase in pull requests merged per week. Escape Meridian interpretation: these results challenge the assumption that better benchmark scores mean more work gets done, but the evidence concerns these model changes and GitHub measures, not every form of coding work.","question":"Would the reported pattern hold for measures of coding work beyond GitHub pull requests?","claims":[{"text":"Hua frames the study as a test of whether higher-scoring coding models lead to more real work getting done.","excerpt":"no study has tested if a model that scores higher on these benchmarks gets more real work done.","support":"source-stated"},{"text":"The study compares GitHub pull requests during a 19-day suspension of access to Claude Fable 5 with pull requests from developers who stayed on Opus 4.8.","excerpt":"We tracked GitHub pull requests of 297 Fable 5 users before, during, and after the suspension. We used 8,908 developers who stayed on Opus 4.8 all the time as the control group.","support":"source-stated"},{"text":"The abstract reports a 1.4-percentage-point increase in the 7-day merge rate associated with access to Fable 5, which was not statistically significant.","excerpt":"increased the 7-day merge rate of pull requests by only 1.4 percentage points [-1.4; +4.3], which was not statistically significant.","support":"source-stated"},{"text":"The abstract reports no significant increase in pull requests merged per week.","excerpt":"We also did not observe a significant increase in the number of pull requests merged per week (+2.4% [-4.9%; +10.2%]).","support":"source-stated"},{"text":"The reported results concern GitHub pull-request measures in the described suspension and model-upgrade comparisons; whether they extend to other measures of coding work remains open.","excerpt":"We tracked GitHub pull requests of 297 Fable 5 users before, during, and after the suspension.","support":"interpretation"}],"limitations":["The evidence supplied is the arXiv abstract, not the full paper; details of the study methods and analyses beyond the abstract are unavailable here.","The reported outcomes are GitHub pull-request merge rates and numbers of merged pull requests; the abstract does not establish how the results apply to other measures of coding work."],"mediaId":"b49f06a805c7d88e1513568277b2f225c91f5eaba0bb1e2600e73f58eaefb0b3","correctionReason":null,"id":"983a6bbd5cbde3fc6ba856e75dbbef08b5700b63639907064d23664b058a9e18","publicationId":"4a975edd-5113-40de-8d9f-4956f09d2354","version":1,"sourceUrl":"https://arxiv.org/abs/2611.23216","sourceTitle":"Higher-Scoring Coding Models Did Not Get More Work Done","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-17","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-18T05:07:08.541Z","createdAt":"2026-11-18T18:33:29.138Z","historical":false,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"b49f06a805c7d88e1513568277b2f225c91f5eaba0bb1e2600e73f58eaefb0b3","kind":"article","url":"https://arxiv.org/abs/2611.23216","sourceUrl":"https://arxiv.org/abs/2611.23216","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-18T18:32:43.769Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.23216","checkedAt":"2026-11-18T18:48:11.778Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"b49f06a805c7d88e1513568277b2f225c91f5eaba0bb1e2600e73f58eaefb0b3","confidence":0.98,"provider":"typesafe","model":"jev-1.13.0","inputHash":"7554b548f069f0a874f56cf1246dc84135d9fef78f678c18208eefe894e9d9f9","decidedAt":"2026-11-18T18:32:44.995Z"},"provenance":{"textHash":"9b3236e2b14c6a0a2785a826354fbbabae559c60aae6474174796c9fd6fcb741","observationId":"55be3cc82753e8b247ead19561800ca49401c96f1f1edf0e577e773f955c2a6c","retrievedAt":"2026-11-18T18:32:43.594Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"7580f6fc-efba-4a38-8259-741a6c13f13a"},"reviewer":{"model":"gpt-6-luna","runId":"0c60be9a-d55f-4a7c-8bed-a221cbe12b1a"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-18T18:33:29.138Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Hua, Yining","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Hua, Yining","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Hua, Yining","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.23216","locator":"head/meta[name=author|citation_author]"},"assertionId":"c9725c53c15665ff21cb57d413cfaa270b0fb983cbb9392847bcdbb488a39d8e:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T18:48:11.778Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c9725c53c15665ff21cb57d413cfaa270b0fb983cbb9392847bcdbb488a39d8e","sourceUrl":"https://arxiv.org/abs/2611.23216","contentSha256":"34f4b00979c01edcbfb0661f345f053fff1504b40ea889e55a7698c20b115b1c","textHash":"9b3236e2b14c6a0a2785a826354fbbabae559c60aae6474174796c9fd6fcb741","observedAt":"2026-11-18T18:48:11.778Z"}}],"participationCoverage":{"total":1,"returned":1,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.23216","snapshotId":"c9725c53c15665ff21cb57d413cfaa270b0fb983cbb9392847bcdbb488a39d8e","contentSha256":"34f4b00979c01edcbfb0661f345f053fff1504b40ea889e55a7698c20b115b1c","textHash":"9b3236e2b14c6a0a2785a826354fbbabae559c60aae6474174796c9fd6fcb741","observedAt":"2026-11-18T18:48:11.778Z","basis":"publisher-page-metadata","sourceLinks":41,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/983a6bbd5cbde3fc6ba856e75dbbef08b5700b63639907064d23664b058a9e18/source"}}}],"audits":[{"publicationId":"4a975edd-5113-40de-8d9f-4956f09d2354","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the study and its framing to author Yining Hua in an arXiv abstract. The paper's primary listing is cs.SE with a cs.AI cross-list, so the cs.AI label is acceptable. Results are presented as what the abstract reports.","excerpt":"Authors:Yining Hua"},{"dimension":"summary","status":"uncertain","explanation":"The summary leaves out why the comparison tests benchmarks at all: most users switched to Opus 4.8, which the source says scores 11 points lower on SWE-Bench Pro. It also omits the cause of the suspension, a U.S. export control directive. Smallest repair: add one sentence covering both points. Separately, the central measures, pull requests and 7-day merge rate, are not explained in plain language. Add tooltips or a brief gloss. The omitted detail that all five upgrade estimates were negative would also preserve the source's force.","excerpt":"most former Fable 5 users switched to Opus 4.8 (11 points lower on SWE-Bench Pro compared to Fable 5)"},{"dimension":"date","status":"pass","explanation":"originalPublished 2026-11-17 matches the v1 submission date shown on the arXiv page.","excerpt":"[Submitted on 17 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"The numeric claims, the non-significance and the study design match the abstract. Saying the effect was 'associated with' access is more cautious than the source's 'increased', not an overstatement. The Escape Meridian interpretation is labeled and scoped to these model changes and GitHub measures. The question follows from the evidence without leading.","excerpt":"no significant increases of merge rates were found with the upgrades (weighted mean: -0.9 percentage points [-1.9; +0.2])"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the original arXiv abstract page, which is the source itself.","excerpt":null}],"createdAt":"2026-11-18T18:33:29.138Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}