{"id":"e0a4acb133aaf6a6ef136a03beaab15e79803488439811e7d0f95b11a2f38267","title":"arXiv cs.AI paper ChainCheck warns benchmark scores may reflect construction artifacts","argument":"The authors of ChainCheck argue that high evidence-sufficiency benchmark scores can reflect artifacts introduced when reasoning chains are broken, rather than multi-hop reasoning. In an exploratory crossed control on 2,214 HotpotQA pairs, a language-model scorer that never reads the chain reached a nominal AUC of 0.997. A one-string match reached the maximum crossed chain-sensitivity statistic, CS4 (0.500); the authors stress that AUC and CS4 are different metrics. They interpret positive CS4 as “operational bridge-consistency sensitivity,” not semantic multi-hop reasoning. The authors report that every evaluated system was sensitive to an extra edit, and model-based naturalness ratings found the edits detectable. But patterns pre-specified on MuSiQue and 2WikiMultiHopQA did not replicate on an independent HotpotQA set: 5 of 6 trained readouts were edit-dominated, and the reference judge’s selectivity interval included zero. Judge scores also depended on batch layout. The paper’s argument is that these scores may capture sensitivity to how examples were constructed rather than the reasoning ability they are meant to measure. External reproduction is partial: the MuSiQue and 2Wiki records are public, but the HotpotQA records and fine-tuned checkpoints are not.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The paper describes the crossed control on 2,214 HotpotQA pairs as exploratory. The HotpotQA records and fine-tuned checkpoints are not public, so external reproduction is partial. The abstract reports model-based naturalness ratings for edit detectability but does not provide their detailed results.","speaker":{"name":"Han, Hyojung, Kang, Ryangkyung, Kim, Youngeun, Jeon, Seung-Hun","roleAtSource":"Article author"},"source":{"title":"ChainCheck: Near-Perfect Evidence-Sufficiency Scores Can Conceal Sensitivity to Construction Artifacts -- Paired Conditional Contrasts that Expose an Evaluation Confound","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.11965","publishedAt":"2026-11-09","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that near-perfect evidence-sufficiency scores may not demonstrate multi-hop reasoning when broken sets are created by editing text.","excerpt":"near-perfect sufficiency scores--and even positive crossed-control scores--need not demonstrate multi-hop reasoning.","url":"https://arxiv.org/abs/2611.11965","relationship":"Source-stated"},{"text":"In an exploratory crossed control, a scorer that never read the chain achieved a nominal AUC of 0.997, while a one-string match reached the maximum CS4 score; the paper says these are different metrics.","excerpt":"a language-model scorer that never reads the chain reaches a nominal AUC of 0.997, and a one-string match reaches the maximum of the crossed chain-sensitivity statistic CS4 (0.500), while the 11 evaluated systems score from 0.308 to 0.484; the two statistics are different metrics.","url":"https://arxiv.org/abs/2611.11965","relationship":"Source-stated"},{"text":"The authors interpret positive CS4 as operational bridge-consistency sensitivity, not semantic multi-hop reasoning.","excerpt":"We therefore read positive CS4 as operational bridge-consistency sensitivity, not as semantic multi-hop reasoning.","url":"https://arxiv.org/abs/2611.11965","relationship":"Source-stated"},{"text":"On an independent HotpotQA set, patterns pre-specified on MuSiQue and 2WikiMultiHopQA did not replicate; most trained readouts were edit-dominated, and the reference judge’s selectivity interval included zero.","excerpt":"On an independent HotpotQA set of 2,276 pairs, patterns pre-specified on MuSiQue and 2WikiMultiHopQA did not replicate: 5 of 6 trained readouts were edit-dominated, and the reference judge's selectivity interval [-0.007, +0.042] includes zero.","url":"https://arxiv.org/abs/2611.11965","relationship":"Source-stated"},{"text":"External reproduction is partial because the HotpotQA records and fine-tuned checkpoints are not public.","excerpt":"The MuSiQue and 2Wiki records are public; the HotpotQA records and the fine-tuned checkpoints are not, so external reproduction is partial.","url":"https://arxiv.org/abs/2611.11965","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.11965","checkedAt":"2026-11-12T08:34:07.246Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Would the authors’ findings replicate if the HotpotQA records and fine-tuned checkpoints were available for external reproduction?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI paper ChainCheck warns benchmark scores may reflect construction artifacts","summary":"The authors of ChainCheck argue that high evidence-sufficiency benchmark scores can reflect artifacts introduced when reasoning chains are broken, rather than multi-hop reasoning. In an exploratory crossed control on 2,214 HotpotQA pairs, a language-model scorer that never reads the chain reached a nominal AUC of 0.997. A one-string match reached the maximum crossed chain-sensitivity statistic, CS4 (0.500); the authors stress that AUC and CS4 are different metrics. They interpret positive CS4 as “operational bridge-consistency sensitivity,” not semantic multi-hop reasoning. The authors report that every evaluated system was sensitive to an extra edit, and model-based naturalness ratings found the edits detectable. But patterns pre-specified on MuSiQue and 2WikiMultiHopQA did not replicate on an independent HotpotQA set: 5 of 6 trained readouts were edit-dominated, and the reference judge’s selectivity interval included zero. Judge scores also depended on batch layout. The paper’s argument is that these scores may capture sensitivity to how examples were constructed rather than the reasoning ability they are meant to measure. External reproduction is partial: the MuSiQue and 2Wiki records are public, but the HotpotQA records and fine-tuned checkpoints are not.","question":"Would the authors’ findings replicate if the HotpotQA records and fine-tuned checkpoints were available for external reproduction?","claims":[{"text":"The authors argue that near-perfect evidence-sufficiency scores may not demonstrate multi-hop reasoning when broken sets are created by editing text.","excerpt":"near-perfect sufficiency scores--and even positive crossed-control scores--need not demonstrate multi-hop reasoning.","support":"source-stated"},{"text":"In an exploratory crossed control, a scorer that never read the chain achieved a nominal AUC of 0.997, while a one-string match reached the maximum CS4 score; the paper says these are different metrics.","excerpt":"a language-model scorer that never reads the chain reaches a nominal AUC of 0.997, and a one-string match reaches the maximum of the crossed chain-sensitivity statistic CS4 (0.500), while the 11 evaluated systems score from 0.308 to 0.484; the two statistics are different metrics.","support":"source-stated"},{"text":"The authors interpret positive CS4 as operational bridge-consistency sensitivity, not semantic multi-hop reasoning.","excerpt":"We therefore read positive CS4 as operational bridge-consistency sensitivity, not as semantic multi-hop reasoning.","support":"source-stated"},{"text":"On an independent HotpotQA set, patterns pre-specified on MuSiQue and 2WikiMultiHopQA did not replicate; most trained readouts were edit-dominated, and the reference judge’s selectivity interval included zero.","excerpt":"On an independent HotpotQA set of 2,276 pairs, patterns pre-specified on MuSiQue and 2WikiMultiHopQA did not replicate: 5 of 6 trained readouts were edit-dominated, and the reference judge's selectivity interval [-0.007, +0.042] includes zero.","support":"source-stated"},{"text":"External reproduction is partial because the HotpotQA records and fine-tuned checkpoints are not public.","excerpt":"The MuSiQue and 2Wiki records are public; the HotpotQA records and the fine-tuned checkpoints are not, so external reproduction is partial.","support":"source-stated"}],"limitations":["The paper describes the crossed control on 2,214 HotpotQA pairs as exploratory.","The HotpotQA records and fine-tuned checkpoints are not public, so external reproduction is partial.","The abstract reports model-based naturalness ratings for edit detectability but does not provide their detailed results."],"mediaId":"592c69146e05518a6fc2a3d8d3036ab860ff3b4708edecf8b5449d932b80e34e","correctionReason":null,"id":"e0a4acb133aaf6a6ef136a03beaab15e79803488439811e7d0f95b11a2f38267","publicationId":"cfc8a74c-64ac-4cb2-8526-b711d93c5b4f","version":1,"sourceUrl":"https://arxiv.org/abs/2611.11965","sourceTitle":"ChainCheck: Near-Perfect Evidence-Sufficiency Scores Can Conceal Sensitivity to Construction Artifacts -- Paired Conditional Contrasts that Expose an Evaluation Confound","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-09","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-10T08:40:00.250Z","createdAt":"2026-11-12T08:03:04.491Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"592c69146e05518a6fc2a3d8d3036ab860ff3b4708edecf8b5449d932b80e34e","kind":"article","url":"https://arxiv.org/abs/2611.11965","sourceUrl":"https://arxiv.org/abs/2611.11965","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T08:02:09.204Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.11965","checkedAt":"2026-11-12T08:34:07.246Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"592c69146e05518a6fc2a3d8d3036ab860ff3b4708edecf8b5449d932b80e34e","confidence":0.56,"provider":"typesafe","model":"jev-1.13.0","inputHash":"07b7bbee777efc9efe6b1dd2d70a51b92467fbb8d89caea90bf1f9a212d6f79a","decidedAt":"2026-11-12T08:02:10.042Z"},"provenance":{"textHash":"ba37af1225bfb5177b4f590bf51eb3f66652a356f1626d0dda8b5f4152863a1b","observationId":"cc4940037d627765c1aefc39820c17798cd6c21caa773c471c2d2b31b39a2a9a","retrievedAt":"2026-11-12T08:02:09.104Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"60e7b044-0be6-4a83-81a9-994910bcb24f"},"reviewer":{"model":"gpt-6-luna","runId":"d9d84b9b-ee6b-45bd-8a18-3efc7d85f634"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-12T08:03:04.491Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Han, Hyojung","url":null,"kind":"unknown"},{"name":"Kang, Ryangkyung","url":null,"kind":"unknown"},{"name":"Kim, Youngeun","url":null,"kind":"unknown"},{"name":"Jeon, Seung-Hun","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Han, Hyojung","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Han, Hyojung","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11965","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e944ac850f41077d2f7cd0fc0e817ff2e9810eafd1f75aac9c609869c181fcb:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T08:34:07.246Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e944ac850f41077d2f7cd0fc0e817ff2e9810eafd1f75aac9c609869c181fcb","sourceUrl":"https://arxiv.org/abs/2611.11965","contentSha256":"f6d0d41b1a8e95b3e9c7781b19b3f6aacbeeb78c2727543d46a901f7b3f20175","textHash":"ba37af1225bfb5177b4f590bf51eb3f66652a356f1626d0dda8b5f4152863a1b","observedAt":"2026-11-12T08:34:07.246Z"}},{"actor":{"name":"Kang, Ryangkyung","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Kang, Ryangkyung","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11965","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e944ac850f41077d2f7cd0fc0e817ff2e9810eafd1f75aac9c609869c181fcb:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T08:34:07.246Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e944ac850f41077d2f7cd0fc0e817ff2e9810eafd1f75aac9c609869c181fcb","sourceUrl":"https://arxiv.org/abs/2611.11965","contentSha256":"f6d0d41b1a8e95b3e9c7781b19b3f6aacbeeb78c2727543d46a901f7b3f20175","textHash":"ba37af1225bfb5177b4f590bf51eb3f66652a356f1626d0dda8b5f4152863a1b","observedAt":"2026-11-12T08:34:07.246Z"}},{"actor":{"name":"Kim, Youngeun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Kim, Youngeun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11965","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e944ac850f41077d2f7cd0fc0e817ff2e9810eafd1f75aac9c609869c181fcb:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T08:34:07.246Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e944ac850f41077d2f7cd0fc0e817ff2e9810eafd1f75aac9c609869c181fcb","sourceUrl":"https://arxiv.org/abs/2611.11965","contentSha256":"f6d0d41b1a8e95b3e9c7781b19b3f6aacbeeb78c2727543d46a901f7b3f20175","textHash":"ba37af1225bfb5177b4f590bf51eb3f66652a356f1626d0dda8b5f4152863a1b","observedAt":"2026-11-12T08:34:07.246Z"}},{"actor":{"name":"Jeon, Seung-Hun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Jeon, Seung-Hun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11965","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e944ac850f41077d2f7cd0fc0e817ff2e9810eafd1f75aac9c609869c181fcb:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T08:34:07.246Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e944ac850f41077d2f7cd0fc0e817ff2e9810eafd1f75aac9c609869c181fcb","sourceUrl":"https://arxiv.org/abs/2611.11965","contentSha256":"f6d0d41b1a8e95b3e9c7781b19b3f6aacbeeb78c2727543d46a901f7b3f20175","textHash":"ba37af1225bfb5177b4f590bf51eb3f66652a356f1626d0dda8b5f4152863a1b","observedAt":"2026-11-12T08:34:07.246Z"}}],"participationCoverage":{"total":4,"returned":4,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.11965","snapshotId":"7e944ac850f41077d2f7cd0fc0e817ff2e9810eafd1f75aac9c609869c181fcb","contentSha256":"f6d0d41b1a8e95b3e9c7781b19b3f6aacbeeb78c2727543d46a901f7b3f20175","textHash":"ba37af1225bfb5177b4f590bf51eb3f66652a356f1626d0dda8b5f4152863a1b","observedAt":"2026-11-12T08:34:07.246Z","basis":"publisher-page-metadata","sourceLinks":42,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/e0a4acb133aaf6a6ef136a03beaab15e79803488439811e7d0f95b11a2f38267/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-paper-chaincheck-warns-benchmark-scores-may-reflect-e0a4acb1","formats":{"html":"/readings/arxiv-cs-ai-paper-chaincheck-warns-benchmark-scores-may-reflect-e0a4acb1","markdown":"/readings/arxiv-cs-ai-paper-chaincheck-warns-benchmark-scores-may-reflect-e0a4acb1.md","json":"/readings/arxiv-cs-ai-paper-chaincheck-warns-benchmark-scores-may-reflect-e0a4acb1.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-fine-tuning","name":"Fine-tuning","explanation":"Further training of an existing model to adapt it to a particular task or use.","referenceUrl":"https://developers.google.com/machine-learning/glossary#fine-tuning"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"e0a4acb133aaf6a6ef136a03beaab15e79803488439811e7d0f95b11a2f38267","revision":"cfc8a74c-64ac-4cb2-8526-b711d93c5b4f","revisionKind":"publication-id","title":"arXiv cs.AI paper ChainCheck warns benchmark scores may reflect construction artifacts","url":"https://escapemeridian.com/readings/arxiv-cs-ai-paper-chaincheck-warns-benchmark-scores-may-reflect-e0a4acb1"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.11965","revision":"ba37af1225bfb5177b4f590bf51eb3f66652a356f1626d0dda8b5f4152863a1b","revisionKind":"sha256-source-text","title":"ChainCheck: Near-Perfect Evidence-Sufficiency Scores Can Conceal Sensitivity to Construction Artifacts -- Paired Conditional Contrasts that Expose an Evaluation Confound","url":"https://arxiv.org/abs/2611.11965"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"e0a4acb133aaf6a6ef136a03beaab15e79803488439811e7d0f95b11a2f38267","title":"arXiv cs.AI paper ChainCheck warns benchmark scores may reflect construction artifacts","argument":"The authors of ChainCheck argue that high evidence-sufficiency benchmark scores can reflect artifacts introduced when reasoning chains are broken, rather than multi-hop reasoning. In an exploratory crossed control on 2,214 HotpotQA pairs, a language-model scorer that never reads the chain reached a nominal AUC of 0.997. A one-string match reached the maximum crossed chain-sensitivity statistic, CS4 (0.500); the authors stress that AUC and CS4 are different metrics. They interpret positive CS4 as “operational bridge-consistency sensitivity,” not semantic multi-hop reasoning. The authors report that every evaluated system was sensitive to an extra edit, and model-based naturalness ratings found the edits detectable. But patterns pre-specified on MuSiQue and 2WikiMultiHopQA did not replicate on an independent HotpotQA set: 5 of 6 trained readouts were edit-dominated, and the reference judge’s selectivity interval included zero. Judge scores also depended on batch layout. The paper’s argument is that these scores may capture sensitivity to how examples were constructed rather than the reasoning ability they are meant to measure. External reproduction is partial: the MuSiQue and 2Wiki records are public, but the HotpotQA records and fine-tuned checkpoints are not.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The paper describes the crossed control on 2,214 HotpotQA pairs as exploratory. The HotpotQA records and fine-tuned checkpoints are not public, so external reproduction is partial. The abstract reports model-based naturalness ratings for edit detectability but does not provide their detailed results.","speaker":{"name":"Han, Hyojung, Kang, Ryangkyung, Kim, Youngeun, Jeon, Seung-Hun","roleAtSource":"Article author"},"source":{"title":"ChainCheck: Near-Perfect Evidence-Sufficiency Scores Can Conceal Sensitivity to Construction Artifacts -- Paired Conditional Contrasts that Expose an Evaluation Confound","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.11965","publishedAt":"2026-11-09","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that near-perfect evidence-sufficiency scores may not demonstrate multi-hop reasoning when broken sets are created by editing text.","excerpt":"near-perfect sufficiency scores--and even positive crossed-control scores--need not demonstrate multi-hop reasoning.","url":"https://arxiv.org/abs/2611.11965","relationship":"Source-stated"},{"text":"In an exploratory crossed control, a scorer that never read the chain achieved a nominal AUC of 0.997, while a one-string match reached the maximum CS4 score; the paper says these are different metrics.","excerpt":"a language-model scorer that never reads the chain reaches a nominal AUC of 0.997, and a one-string match reaches the maximum of the crossed chain-sensitivity statistic CS4 (0.500), while the 11 evaluated systems score from 0.308 to 0.484; the two statistics are different metrics.","url":"https://arxiv.org/abs/2611.11965","relationship":"Source-stated"},{"text":"The authors interpret positive CS4 as operational bridge-consistency sensitivity, not semantic multi-hop reasoning.","excerpt":"We therefore read positive CS4 as operational bridge-consistency sensitivity, not as semantic multi-hop reasoning.","url":"https://arxiv.org/abs/2611.11965","relationship":"Source-stated"},{"text":"On an independent HotpotQA set, patterns pre-specified on MuSiQue and 2WikiMultiHopQA did not replicate; most trained readouts were edit-dominated, and the reference judge’s selectivity interval included zero.","excerpt":"On an independent HotpotQA set of 2,276 pairs, patterns pre-specified on MuSiQue and 2WikiMultiHopQA did not replicate: 5 of 6 trained readouts were edit-dominated, and the reference judge's selectivity interval [-0.007, +0.042] includes zero.","url":"https://arxiv.org/abs/2611.11965","relationship":"Source-stated"},{"text":"External reproduction is partial because the HotpotQA records and fine-tuned checkpoints are not public.","excerpt":"The MuSiQue and 2Wiki records are public; the HotpotQA records and the fine-tuned checkpoints are not, so external reproduction is partial.","url":"https://arxiv.org/abs/2611.11965","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.11965","checkedAt":"2026-11-12T08:34:07.246Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Would the authors’ findings replicate if the HotpotQA records and fine-tuned checkpoints were available for external reproduction?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI paper ChainCheck warns benchmark scores may reflect construction artifacts","summary":"The authors of ChainCheck argue that high evidence-sufficiency benchmark scores can reflect artifacts introduced when reasoning chains are broken, rather than multi-hop reasoning. In an exploratory crossed control on 2,214 HotpotQA pairs, a language-model scorer that never reads the chain reached a nominal AUC of 0.997. A one-string match reached the maximum crossed chain-sensitivity statistic, CS4 (0.500); the authors stress that AUC and CS4 are different metrics. They interpret positive CS4 as “operational bridge-consistency sensitivity,” not semantic multi-hop reasoning. The authors report that every evaluated system was sensitive to an extra edit, and model-based naturalness ratings found the edits detectable. But patterns pre-specified on MuSiQue and 2WikiMultiHopQA did not replicate on an independent HotpotQA set: 5 of 6 trained readouts were edit-dominated, and the reference judge’s selectivity interval included zero. Judge scores also depended on batch layout. The paper’s argument is that these scores may capture sensitivity to how examples were constructed rather than the reasoning ability they are meant to measure. External reproduction is partial: the MuSiQue and 2Wiki records are public, but the HotpotQA records and fine-tuned checkpoints are not.","question":"Would the authors’ findings replicate if the HotpotQA records and fine-tuned checkpoints were available for external reproduction?","claims":[{"text":"The authors argue that near-perfect evidence-sufficiency scores may not demonstrate multi-hop reasoning when broken sets are created by editing text.","excerpt":"near-perfect sufficiency scores--and even positive crossed-control scores--need not demonstrate multi-hop reasoning.","support":"source-stated"},{"text":"In an exploratory crossed control, a scorer that never read the chain achieved a nominal AUC of 0.997, while a one-string match reached the maximum CS4 score; the paper says these are different metrics.","excerpt":"a language-model scorer that never reads the chain reaches a nominal AUC of 0.997, and a one-string match reaches the maximum of the crossed chain-sensitivity statistic CS4 (0.500), while the 11 evaluated systems score from 0.308 to 0.484; the two statistics are different metrics.","support":"source-stated"},{"text":"The authors interpret positive CS4 as operational bridge-consistency sensitivity, not semantic multi-hop reasoning.","excerpt":"We therefore read positive CS4 as operational bridge-consistency sensitivity, not as semantic multi-hop reasoning.","support":"source-stated"},{"text":"On an independent HotpotQA set, patterns pre-specified on MuSiQue and 2WikiMultiHopQA did not replicate; most trained readouts were edit-dominated, and the reference judge’s selectivity interval included zero.","excerpt":"On an independent HotpotQA set of 2,276 pairs, patterns pre-specified on MuSiQue and 2WikiMultiHopQA did not replicate: 5 of 6 trained readouts were edit-dominated, and the reference judge's selectivity interval [-0.007, +0.042] includes zero.","support":"source-stated"},{"text":"External reproduction is partial because the HotpotQA records and fine-tuned checkpoints are not public.","excerpt":"The MuSiQue and 2Wiki records are public; the HotpotQA records and the fine-tuned checkpoints are not, so external reproduction is partial.","support":"source-stated"}],"limitations":["The paper describes the crossed control on 2,214 HotpotQA pairs as exploratory.","The HotpotQA records and fine-tuned checkpoints are not public, so external reproduction is partial.","The abstract reports model-based naturalness ratings for edit detectability but does not provide their detailed results."],"mediaId":"592c69146e05518a6fc2a3d8d3036ab860ff3b4708edecf8b5449d932b80e34e","correctionReason":null,"id":"e0a4acb133aaf6a6ef136a03beaab15e79803488439811e7d0f95b11a2f38267","publicationId":"cfc8a74c-64ac-4cb2-8526-b711d93c5b4f","version":1,"sourceUrl":"https://arxiv.org/abs/2611.11965","sourceTitle":"ChainCheck: Near-Perfect Evidence-Sufficiency Scores Can Conceal Sensitivity to Construction Artifacts -- Paired Conditional Contrasts that Expose an Evaluation Confound","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-09","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-10T08:40:00.250Z","createdAt":"2026-11-12T08:03:04.491Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"592c69146e05518a6fc2a3d8d3036ab860ff3b4708edecf8b5449d932b80e34e","kind":"article","url":"https://arxiv.org/abs/2611.11965","sourceUrl":"https://arxiv.org/abs/2611.11965","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T08:02:09.204Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.11965","checkedAt":"2026-11-12T08:34:07.246Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"592c69146e05518a6fc2a3d8d3036ab860ff3b4708edecf8b5449d932b80e34e","confidence":0.56,"provider":"typesafe","model":"jev-1.13.0","inputHash":"07b7bbee777efc9efe6b1dd2d70a51b92467fbb8d89caea90bf1f9a212d6f79a","decidedAt":"2026-11-12T08:02:10.042Z"},"provenance":{"textHash":"ba37af1225bfb5177b4f590bf51eb3f66652a356f1626d0dda8b5f4152863a1b","observationId":"cc4940037d627765c1aefc39820c17798cd6c21caa773c471c2d2b31b39a2a9a","retrievedAt":"2026-11-12T08:02:09.104Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"60e7b044-0be6-4a83-81a9-994910bcb24f"},"reviewer":{"model":"gpt-6-luna","runId":"d9d84b9b-ee6b-45bd-8a18-3efc7d85f634"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-12T08:03:04.491Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Han, Hyojung","url":null,"kind":"unknown"},{"name":"Kang, Ryangkyung","url":null,"kind":"unknown"},{"name":"Kim, Youngeun","url":null,"kind":"unknown"},{"name":"Jeon, Seung-Hun","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Han, Hyojung","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Han, Hyojung","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11965","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e944ac850f41077d2f7cd0fc0e817ff2e9810eafd1f75aac9c609869c181fcb:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T08:34:07.246Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e944ac850f41077d2f7cd0fc0e817ff2e9810eafd1f75aac9c609869c181fcb","sourceUrl":"https://arxiv.org/abs/2611.11965","contentSha256":"f6d0d41b1a8e95b3e9c7781b19b3f6aacbeeb78c2727543d46a901f7b3f20175","textHash":"ba37af1225bfb5177b4f590bf51eb3f66652a356f1626d0dda8b5f4152863a1b","observedAt":"2026-11-12T08:34:07.246Z"}},{"actor":{"name":"Kang, Ryangkyung","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Kang, Ryangkyung","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11965","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e944ac850f41077d2f7cd0fc0e817ff2e9810eafd1f75aac9c609869c181fcb:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T08:34:07.246Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e944ac850f41077d2f7cd0fc0e817ff2e9810eafd1f75aac9c609869c181fcb","sourceUrl":"https://arxiv.org/abs/2611.11965","contentSha256":"f6d0d41b1a8e95b3e9c7781b19b3f6aacbeeb78c2727543d46a901f7b3f20175","textHash":"ba37af1225bfb5177b4f590bf51eb3f66652a356f1626d0dda8b5f4152863a1b","observedAt":"2026-11-12T08:34:07.246Z"}},{"actor":{"name":"Kim, Youngeun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Kim, Youngeun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11965","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e944ac850f41077d2f7cd0fc0e817ff2e9810eafd1f75aac9c609869c181fcb:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T08:34:07.246Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e944ac850f41077d2f7cd0fc0e817ff2e9810eafd1f75aac9c609869c181fcb","sourceUrl":"https://arxiv.org/abs/2611.11965","contentSha256":"f6d0d41b1a8e95b3e9c7781b19b3f6aacbeeb78c2727543d46a901f7b3f20175","textHash":"ba37af1225bfb5177b4f590bf51eb3f66652a356f1626d0dda8b5f4152863a1b","observedAt":"2026-11-12T08:34:07.246Z"}},{"actor":{"name":"Jeon, Seung-Hun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Jeon, Seung-Hun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11965","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e944ac850f41077d2f7cd0fc0e817ff2e9810eafd1f75aac9c609869c181fcb:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T08:34:07.246Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e944ac850f41077d2f7cd0fc0e817ff2e9810eafd1f75aac9c609869c181fcb","sourceUrl":"https://arxiv.org/abs/2611.11965","contentSha256":"f6d0d41b1a8e95b3e9c7781b19b3f6aacbeeb78c2727543d46a901f7b3f20175","textHash":"ba37af1225bfb5177b4f590bf51eb3f66652a356f1626d0dda8b5f4152863a1b","observedAt":"2026-11-12T08:34:07.246Z"}}],"participationCoverage":{"total":4,"returned":4,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.11965","snapshotId":"7e944ac850f41077d2f7cd0fc0e817ff2e9810eafd1f75aac9c609869c181fcb","contentSha256":"f6d0d41b1a8e95b3e9c7781b19b3f6aacbeeb78c2727543d46a901f7b3f20175","textHash":"ba37af1225bfb5177b4f590bf51eb3f66652a356f1626d0dda8b5f4152863a1b","observedAt":"2026-11-12T08:34:07.246Z","basis":"publisher-page-metadata","sourceLinks":42,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/e0a4acb133aaf6a6ef136a03beaab15e79803488439811e7d0f95b11a2f38267/source"}}}],"audits":[{"publicationId":"cfc8a74c-64ac-4cb2-8526-b711d93c5b4f","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading consistently attributes the findings and interpretations to the ChainCheck authors and the arXiv cs.AI paper. It does not present them as established facts or as Escape Meridian conclusions. The named authors and venue match the source.","excerpt":"Authors:Hyojung Han, Ryangkyung Kang, Youngeun Kim, Seung-Hun Jeon"},{"dimension":"summary","status":"uncertain","explanation":"The summary is faithful, but several central terms are left unexplained for a general reader: evidence-sufficiency benchmark, multi-hop reasoning, crossed control, AUC, CS4, HotpotQA, trained readouts and selectivity interval. A short plain-language opening would fix this, such as: benchmarks test whether a model can tell a complete set of passages from one with a missing link. The summary also drops the comparison that gives the finding its force: all 11 real systems scored 0.308 to 0.484 on CS4, while a trivial one-string match hit the maximum of 0.500. Adding that clause restores the stakes.","excerpt":"while the 11 evaluated systems score from 0.308 to 0.484"},{"dimension":"date","status":"pass","explanation":"The originalPublished date of 2026-11-09 matches the arXiv v1 submission date. The feed date of November 10 was correctly not used.","excerpt":"[Submitted on 9 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"Each claim matches its verbatim excerpt. The reading keeps the source's caveats: the crossed control was exploratory, AUC and CS4 are different metrics, and external reproduction is only partial. It does not add invented benefits or costs. The question about whether the findings would replicate with the withheld data follows directly from the paper's own reproducibility statement.","excerpt":"the HotpotQA records and the fine-tuned checkpoints are not, so external reproduction is partial."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the arXiv abstract page, which is the original source.","excerpt":null}],"createdAt":"2026-11-12T08:03:04.491Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}