{"id":"26d7816600ecc2026b64f973b52d14baf2d6a332e1d0d2e3cbada3fc238269db","title":"arXiv cs.AI: Yiwei Qin and coauthors argue benchleaking is reaching pre-training","argument":"In an arXiv paper, Yiwei Qin and coauthors call “benchleaking” the entry of a benchmark’s own test examples into training data, where a score may then reflect recall rather than capability. They distinguish it from “benchmaxxing,” which optimizes toward a benchmark without seeing its test set. The authors cite audits finding explicit traces in post-training; they also report the problem has reached pre-training. In few-shot continuation tests across 20 benchmarks and four public base models, some models produced test questions and answers absent from their input. They report large differences between models, including leakage of HumanEval problems with reference implementations and OpenBookQA questions with answers in ERNIE-5.5-Flash-Base. To estimate how much scores depend on leaked examples, they define an Anti-Hack score that marks each leaked example wrong while keeping the number of examples unchanged. Their suite-wide averages range from 1.3 points for Hy3-preview-Base to 16.6 for ERNIE-5.5-Flash-Base. Because the authors describe open base models as a foundation for downstream research, they argue benchleaking reaches beyond leaderboard rankings and call for decontamination reports and contamination-adjusted scores to become standard practice.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported pre-training results cover few-shot continuation on 20 benchmarks and four public base models; how consistently they generalize beyond that tested suite remains unresolved. The abstract summarizes the findings but does not provide the full methods or evidence needed here to assess how leakage was detected across the suite.","speaker":{"name":"Qin, Yiwei, Xie, Muhang, Li, Xuefeng, Jiang, Mohan, Fu, Dayuan, Mi, Tiantian, Guo, Qipeng, Liu, Pengfei","roleAtSource":"Article author"},"source":{"title":"Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.21315","publishedAt":"2026-11-16","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors define benchleaking as benchmark test examples entering training data, potentially making a score reflect recall rather than capability, and distinguish it from benchmaxxing.","excerpt":"We use the term benchleaking for the case in which a benchmark's own test examples enter training data, so that a score reflects recall rather than capability. This differs from benchmaxxing","url":"https://arxiv.org/abs/2611.21315","relationship":"Source-stated"},{"text":"The authors tested few-shot continuation on 20 benchmarks across four public base models and report that some models produced other test examples absent from their input.","excerpt":"Using few-shot continuation on 20 benchmarks across four public base models, we find that some base models write out other test examples---complete questions, options, and answers---that appear nowhere in their input.","url":"https://arxiv.org/abs/2611.21315","relationship":"Source-stated"},{"text":"The authors report that 96% of HumanEval problems and 85% of OpenBookQA questions leak in ERNIE-5.5-Flash-Base, compared with 9% and none in DeepSeek-V4-Flash-Base.","excerpt":"96% of HumanEval problems leak with their reference implementations and 85% of OpenBookQA questions leak with their answers, against 9% and none for DeepSeek-V4-Flash-Base.","url":"https://arxiv.org/abs/2611.21315","relationship":"Source-stated"},{"text":"The authors define an Anti-Hack score that counts leaked examples as wrong while leaving the number of examples unchanged; suite-wide average costs range from 1.3 to 16.6 points among the models cited.","excerpt":"Averaged over the suite, it costs DeepSeek-V4-Flash-Base 1.5 points and Hy3-preview-Base 1.3, against 5.0 for MiMo-V2.5-Base and 16.6 for ERNIE-5.5-Flash-Base.","url":"https://arxiv.org/abs/2611.21315","relationship":"Source-stated"},{"text":"The authors argue benchleaking matters beyond leaderboard rankings because open base models underpin downstream research, and call for decontamination reports and contamination-adjusted scores as standard practice.","excerpt":"Open base models are the foundation of much downstream research, so benchleaking now affects more than leaderboard rankings. We call for decontamination reports and contamination-adjusted scores to become standard practice.","url":"https://arxiv.org/abs/2611.21315","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.21315","checkedAt":"2026-11-18T21:48:09.936Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How consistently would the reported leakage and Anti-Hack score differences hold across benchmarks and base models beyond the tested suite?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Yiwei Qin and coauthors argue benchleaking is reaching pre-training","summary":"In an arXiv paper, Yiwei Qin and coauthors call “benchleaking” the entry of a benchmark’s own test examples into training data, where a score may then reflect recall rather than capability. They distinguish it from “benchmaxxing,” which optimizes toward a benchmark without seeing its test set. The authors cite audits finding explicit traces in post-training; they also report the problem has reached pre-training. In few-shot continuation tests across 20 benchmarks and four public base models, some models produced test questions and answers absent from their input. They report large differences between models, including leakage of HumanEval problems with reference implementations and OpenBookQA questions with answers in ERNIE-5.5-Flash-Base. To estimate how much scores depend on leaked examples, they define an Anti-Hack score that marks each leaked example wrong while keeping the number of examples unchanged. Their suite-wide averages range from 1.3 points for Hy3-preview-Base to 16.6 for ERNIE-5.5-Flash-Base. Because the authors describe open base models as a foundation for downstream research, they argue benchleaking reaches beyond leaderboard rankings and call for decontamination reports and contamination-adjusted scores to become standard practice.","question":"How consistently would the reported leakage and Anti-Hack score differences hold across benchmarks and base models beyond the tested suite?","claims":[{"text":"The authors define benchleaking as benchmark test examples entering training data, potentially making a score reflect recall rather than capability, and distinguish it from benchmaxxing.","excerpt":"We use the term benchleaking for the case in which a benchmark's own test examples enter training data, so that a score reflects recall rather than capability. This differs from benchmaxxing","support":"source-stated"},{"text":"The authors tested few-shot continuation on 20 benchmarks across four public base models and report that some models produced other test examples absent from their input.","excerpt":"Using few-shot continuation on 20 benchmarks across four public base models, we find that some base models write out other test examples---complete questions, options, and answers---that appear nowhere in their input.","support":"source-stated"},{"text":"The authors report that 96% of HumanEval problems and 85% of OpenBookQA questions leak in ERNIE-5.5-Flash-Base, compared with 9% and none in DeepSeek-V4-Flash-Base.","excerpt":"96% of HumanEval problems leak with their reference implementations and 85% of OpenBookQA questions leak with their answers, against 9% and none for DeepSeek-V4-Flash-Base.","support":"source-stated"},{"text":"The authors define an Anti-Hack score that counts leaked examples as wrong while leaving the number of examples unchanged; suite-wide average costs range from 1.3 to 16.6 points among the models cited.","excerpt":"Averaged over the suite, it costs DeepSeek-V4-Flash-Base 1.5 points and Hy3-preview-Base 1.3, against 5.0 for MiMo-V2.5-Base and 16.6 for ERNIE-5.5-Flash-Base.","support":"source-stated"},{"text":"The authors argue benchleaking matters beyond leaderboard rankings because open base models underpin downstream research, and call for decontamination reports and contamination-adjusted scores as standard practice.","excerpt":"Open base models are the foundation of much downstream research, so benchleaking now affects more than leaderboard rankings. We call for decontamination reports and contamination-adjusted scores to become standard practice.","support":"source-stated"}],"limitations":["The reported pre-training results cover few-shot continuation on 20 benchmarks and four public base models; how consistently they generalize beyond that tested suite remains unresolved.","The abstract summarizes the findings but does not provide the full methods or evidence needed here to assess how leakage was detected across the suite."],"mediaId":"2dc3416471ca08d51c1bd4efc9ce72fb8e36e063881051a85f39d0153e0a0ef3","correctionReason":null,"id":"26d7816600ecc2026b64f973b52d14baf2d6a332e1d0d2e3cbada3fc238269db","publicationId":"fdbf4c41-a9fd-45a9-810c-1feaad7e9f5f","version":1,"sourceUrl":"https://arxiv.org/abs/2611.21315","sourceTitle":"Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-16","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-17T08:31:39.138Z","createdAt":"2026-11-18T21:17:07.200Z","historical":false,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"2dc3416471ca08d51c1bd4efc9ce72fb8e36e063881051a85f39d0153e0a0ef3","kind":"article","url":"https://arxiv.org/abs/2611.21315","sourceUrl":"https://arxiv.org/abs/2611.21315","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-18T21:16:09.563Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.21315","checkedAt":"2026-11-18T21:48:09.936Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"2dc3416471ca08d51c1bd4efc9ce72fb8e36e063881051a85f39d0153e0a0ef3","confidence":0.77,"provider":"typesafe","model":"jev-1.13.0","inputHash":"6348705b602797ab5f4baf08031aefc16dbf8279ffbbf97b4f7a7dbba20f83b9","decidedAt":"2026-11-18T21:16:10.419Z"},"provenance":{"textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observationId":"83f2b73c0a6f188030a7aaeb3ce1270ce397fd78b5629f084f13771e5c5d1167","retrievedAt":"2026-11-18T21:16:09.474Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"d10a7c96-5856-4773-8d5c-ebfce1411d2b"},"reviewer":{"model":"gpt-6-luna","runId":"37682276-e202-44b6-8839-06dca88ec5a4"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-18T21:17:07.200Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Qin, Yiwei","url":null,"kind":"unknown"},{"name":"Xie, Muhang","url":null,"kind":"unknown"},{"name":"Li, Xuefeng","url":null,"kind":"unknown"},{"name":"Jiang, Mohan","url":null,"kind":"unknown"},{"name":"Fu, Dayuan","url":null,"kind":"unknown"},{"name":"Mi, Tiantian","url":null,"kind":"unknown"},{"name":"Guo, Qipeng","url":null,"kind":"unknown"},{"name":"Liu, Pengfei","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Qin, Yiwei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Qin, Yiwei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21315","locator":"head/meta[name=author|citation_author]"},"assertionId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T21:48:09.936Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6","sourceUrl":"https://arxiv.org/abs/2611.21315","contentSha256":"5441256af85ca1e16ca4b24bc4345ce7fcbcaf8dca36ef2c187978832695fd0c","textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observedAt":"2026-11-18T21:48:09.936Z"}},{"actor":{"name":"Xie, Muhang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Xie, Muhang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21315","locator":"head/meta[name=author|citation_author]"},"assertionId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T21:48:09.936Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6","sourceUrl":"https://arxiv.org/abs/2611.21315","contentSha256":"5441256af85ca1e16ca4b24bc4345ce7fcbcaf8dca36ef2c187978832695fd0c","textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observedAt":"2026-11-18T21:48:09.936Z"}},{"actor":{"name":"Li, Xuefeng","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Xuefeng","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21315","locator":"head/meta[name=author|citation_author]"},"assertionId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T21:48:09.936Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6","sourceUrl":"https://arxiv.org/abs/2611.21315","contentSha256":"5441256af85ca1e16ca4b24bc4345ce7fcbcaf8dca36ef2c187978832695fd0c","textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observedAt":"2026-11-18T21:48:09.936Z"}},{"actor":{"name":"Jiang, Mohan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Jiang, Mohan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21315","locator":"head/meta[name=author|citation_author]"},"assertionId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T21:48:09.936Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6","sourceUrl":"https://arxiv.org/abs/2611.21315","contentSha256":"5441256af85ca1e16ca4b24bc4345ce7fcbcaf8dca36ef2c187978832695fd0c","textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observedAt":"2026-11-18T21:48:09.936Z"}},{"actor":{"name":"Fu, Dayuan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Fu, Dayuan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21315","locator":"head/meta[name=author|citation_author]"},"assertionId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T21:48:09.936Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6","sourceUrl":"https://arxiv.org/abs/2611.21315","contentSha256":"5441256af85ca1e16ca4b24bc4345ce7fcbcaf8dca36ef2c187978832695fd0c","textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observedAt":"2026-11-18T21:48:09.936Z"}},{"actor":{"name":"Mi, Tiantian","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mi, Tiantian","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21315","locator":"head/meta[name=author|citation_author]"},"assertionId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T21:48:09.936Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6","sourceUrl":"https://arxiv.org/abs/2611.21315","contentSha256":"5441256af85ca1e16ca4b24bc4345ce7fcbcaf8dca36ef2c187978832695fd0c","textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observedAt":"2026-11-18T21:48:09.936Z"}},{"actor":{"name":"Guo, Qipeng","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Guo, Qipeng","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21315","locator":"head/meta[name=author|citation_author]"},"assertionId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T21:48:09.936Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6","sourceUrl":"https://arxiv.org/abs/2611.21315","contentSha256":"5441256af85ca1e16ca4b24bc4345ce7fcbcaf8dca36ef2c187978832695fd0c","textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observedAt":"2026-11-18T21:48:09.936Z"}},{"actor":{"name":"Liu, Pengfei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liu, Pengfei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21315","locator":"head/meta[name=author|citation_author]"},"assertionId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T21:48:09.936Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6","sourceUrl":"https://arxiv.org/abs/2611.21315","contentSha256":"5441256af85ca1e16ca4b24bc4345ce7fcbcaf8dca36ef2c187978832695fd0c","textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observedAt":"2026-11-18T21:48:09.936Z"}}],"participationCoverage":{"total":8,"returned":8,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.21315","snapshotId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6","contentSha256":"5441256af85ca1e16ca4b24bc4345ce7fcbcaf8dca36ef2c187978832695fd0c","textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observedAt":"2026-11-18T21:48:09.936Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/26d7816600ecc2026b64f973b52d14baf2d6a332e1d0d2e3cbada3fc238269db/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-yiwei-qin-and-coauthors-argue-benchleaking-is-26d78166","formats":{"html":"/readings/arxiv-cs-ai-yiwei-qin-and-coauthors-argue-benchleaking-is-26d78166","markdown":"/readings/arxiv-cs-ai-yiwei-qin-and-coauthors-argue-benchleaking-is-26d78166.md","json":"/readings/arxiv-cs-ai-yiwei-qin-and-coauthors-argue-benchleaking-is-26d78166.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-pre-training","name":"Pre-training","explanation":"A model's initial training on a large dataset, before further adaptation for particular uses.","referenceUrl":"https://developers.google.com/machine-learning/glossary#pre-training"},{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-held-out-set","name":"Held-out set","explanation":"Data kept aside during training and used only to test how well a model works on examples it has not seen.","referenceUrl":"https://developers.google.com/machine-learning/glossary#holdout-data"},{"id":"term-zero-shot","name":"Zero-shot","explanation":"Doing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).","referenceUrl":"https://developers.google.com/machine-learning/glossary#zero-shot-prompting"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"26d7816600ecc2026b64f973b52d14baf2d6a332e1d0d2e3cbada3fc238269db","revision":"fdbf4c41-a9fd-45a9-810c-1feaad7e9f5f","revisionKind":"publication-id","title":"arXiv cs.AI: Yiwei Qin and coauthors argue benchleaking is reaching pre-training","url":"https://escapemeridian.com/readings/arxiv-cs-ai-yiwei-qin-and-coauthors-argue-benchleaking-is-26d78166"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.21315","revision":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","revisionKind":"sha256-source-text","title":"Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models","url":"https://arxiv.org/abs/2611.21315"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"26d7816600ecc2026b64f973b52d14baf2d6a332e1d0d2e3cbada3fc238269db","title":"arXiv cs.AI: Yiwei Qin and coauthors argue benchleaking is reaching pre-training","argument":"In an arXiv paper, Yiwei Qin and coauthors call “benchleaking” the entry of a benchmark’s own test examples into training data, where a score may then reflect recall rather than capability. They distinguish it from “benchmaxxing,” which optimizes toward a benchmark without seeing its test set. The authors cite audits finding explicit traces in post-training; they also report the problem has reached pre-training. In few-shot continuation tests across 20 benchmarks and four public base models, some models produced test questions and answers absent from their input. They report large differences between models, including leakage of HumanEval problems with reference implementations and OpenBookQA questions with answers in ERNIE-5.5-Flash-Base. To estimate how much scores depend on leaked examples, they define an Anti-Hack score that marks each leaked example wrong while keeping the number of examples unchanged. Their suite-wide averages range from 1.3 points for Hy3-preview-Base to 16.6 for ERNIE-5.5-Flash-Base. Because the authors describe open base models as a foundation for downstream research, they argue benchleaking reaches beyond leaderboard rankings and call for decontamination reports and contamination-adjusted scores to become standard practice.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported pre-training results cover few-shot continuation on 20 benchmarks and four public base models; how consistently they generalize beyond that tested suite remains unresolved. The abstract summarizes the findings but does not provide the full methods or evidence needed here to assess how leakage was detected across the suite.","speaker":{"name":"Qin, Yiwei, Xie, Muhang, Li, Xuefeng, Jiang, Mohan, Fu, Dayuan, Mi, Tiantian, Guo, Qipeng, Liu, Pengfei","roleAtSource":"Article author"},"source":{"title":"Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.21315","publishedAt":"2026-11-16","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors define benchleaking as benchmark test examples entering training data, potentially making a score reflect recall rather than capability, and distinguish it from benchmaxxing.","excerpt":"We use the term benchleaking for the case in which a benchmark's own test examples enter training data, so that a score reflects recall rather than capability. This differs from benchmaxxing","url":"https://arxiv.org/abs/2611.21315","relationship":"Source-stated"},{"text":"The authors tested few-shot continuation on 20 benchmarks across four public base models and report that some models produced other test examples absent from their input.","excerpt":"Using few-shot continuation on 20 benchmarks across four public base models, we find that some base models write out other test examples---complete questions, options, and answers---that appear nowhere in their input.","url":"https://arxiv.org/abs/2611.21315","relationship":"Source-stated"},{"text":"The authors report that 96% of HumanEval problems and 85% of OpenBookQA questions leak in ERNIE-5.5-Flash-Base, compared with 9% and none in DeepSeek-V4-Flash-Base.","excerpt":"96% of HumanEval problems leak with their reference implementations and 85% of OpenBookQA questions leak with their answers, against 9% and none for DeepSeek-V4-Flash-Base.","url":"https://arxiv.org/abs/2611.21315","relationship":"Source-stated"},{"text":"The authors define an Anti-Hack score that counts leaked examples as wrong while leaving the number of examples unchanged; suite-wide average costs range from 1.3 to 16.6 points among the models cited.","excerpt":"Averaged over the suite, it costs DeepSeek-V4-Flash-Base 1.5 points and Hy3-preview-Base 1.3, against 5.0 for MiMo-V2.5-Base and 16.6 for ERNIE-5.5-Flash-Base.","url":"https://arxiv.org/abs/2611.21315","relationship":"Source-stated"},{"text":"The authors argue benchleaking matters beyond leaderboard rankings because open base models underpin downstream research, and call for decontamination reports and contamination-adjusted scores as standard practice.","excerpt":"Open base models are the foundation of much downstream research, so benchleaking now affects more than leaderboard rankings. We call for decontamination reports and contamination-adjusted scores to become standard practice.","url":"https://arxiv.org/abs/2611.21315","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.21315","checkedAt":"2026-11-18T21:48:09.936Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How consistently would the reported leakage and Anti-Hack score differences hold across benchmarks and base models beyond the tested suite?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Yiwei Qin and coauthors argue benchleaking is reaching pre-training","summary":"In an arXiv paper, Yiwei Qin and coauthors call “benchleaking” the entry of a benchmark’s own test examples into training data, where a score may then reflect recall rather than capability. They distinguish it from “benchmaxxing,” which optimizes toward a benchmark without seeing its test set. The authors cite audits finding explicit traces in post-training; they also report the problem has reached pre-training. In few-shot continuation tests across 20 benchmarks and four public base models, some models produced test questions and answers absent from their input. They report large differences between models, including leakage of HumanEval problems with reference implementations and OpenBookQA questions with answers in ERNIE-5.5-Flash-Base. To estimate how much scores depend on leaked examples, they define an Anti-Hack score that marks each leaked example wrong while keeping the number of examples unchanged. Their suite-wide averages range from 1.3 points for Hy3-preview-Base to 16.6 for ERNIE-5.5-Flash-Base. Because the authors describe open base models as a foundation for downstream research, they argue benchleaking reaches beyond leaderboard rankings and call for decontamination reports and contamination-adjusted scores to become standard practice.","question":"How consistently would the reported leakage and Anti-Hack score differences hold across benchmarks and base models beyond the tested suite?","claims":[{"text":"The authors define benchleaking as benchmark test examples entering training data, potentially making a score reflect recall rather than capability, and distinguish it from benchmaxxing.","excerpt":"We use the term benchleaking for the case in which a benchmark's own test examples enter training data, so that a score reflects recall rather than capability. This differs from benchmaxxing","support":"source-stated"},{"text":"The authors tested few-shot continuation on 20 benchmarks across four public base models and report that some models produced other test examples absent from their input.","excerpt":"Using few-shot continuation on 20 benchmarks across four public base models, we find that some base models write out other test examples---complete questions, options, and answers---that appear nowhere in their input.","support":"source-stated"},{"text":"The authors report that 96% of HumanEval problems and 85% of OpenBookQA questions leak in ERNIE-5.5-Flash-Base, compared with 9% and none in DeepSeek-V4-Flash-Base.","excerpt":"96% of HumanEval problems leak with their reference implementations and 85% of OpenBookQA questions leak with their answers, against 9% and none for DeepSeek-V4-Flash-Base.","support":"source-stated"},{"text":"The authors define an Anti-Hack score that counts leaked examples as wrong while leaving the number of examples unchanged; suite-wide average costs range from 1.3 to 16.6 points among the models cited.","excerpt":"Averaged over the suite, it costs DeepSeek-V4-Flash-Base 1.5 points and Hy3-preview-Base 1.3, against 5.0 for MiMo-V2.5-Base and 16.6 for ERNIE-5.5-Flash-Base.","support":"source-stated"},{"text":"The authors argue benchleaking matters beyond leaderboard rankings because open base models underpin downstream research, and call for decontamination reports and contamination-adjusted scores as standard practice.","excerpt":"Open base models are the foundation of much downstream research, so benchleaking now affects more than leaderboard rankings. We call for decontamination reports and contamination-adjusted scores to become standard practice.","support":"source-stated"}],"limitations":["The reported pre-training results cover few-shot continuation on 20 benchmarks and four public base models; how consistently they generalize beyond that tested suite remains unresolved.","The abstract summarizes the findings but does not provide the full methods or evidence needed here to assess how leakage was detected across the suite."],"mediaId":"2dc3416471ca08d51c1bd4efc9ce72fb8e36e063881051a85f39d0153e0a0ef3","correctionReason":null,"id":"26d7816600ecc2026b64f973b52d14baf2d6a332e1d0d2e3cbada3fc238269db","publicationId":"fdbf4c41-a9fd-45a9-810c-1feaad7e9f5f","version":1,"sourceUrl":"https://arxiv.org/abs/2611.21315","sourceTitle":"Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-16","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-17T08:31:39.138Z","createdAt":"2026-11-18T21:17:07.200Z","historical":false,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"2dc3416471ca08d51c1bd4efc9ce72fb8e36e063881051a85f39d0153e0a0ef3","kind":"article","url":"https://arxiv.org/abs/2611.21315","sourceUrl":"https://arxiv.org/abs/2611.21315","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-18T21:16:09.563Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.21315","checkedAt":"2026-11-18T21:48:09.936Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"2dc3416471ca08d51c1bd4efc9ce72fb8e36e063881051a85f39d0153e0a0ef3","confidence":0.77,"provider":"typesafe","model":"jev-1.13.0","inputHash":"6348705b602797ab5f4baf08031aefc16dbf8279ffbbf97b4f7a7dbba20f83b9","decidedAt":"2026-11-18T21:16:10.419Z"},"provenance":{"textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observationId":"83f2b73c0a6f188030a7aaeb3ce1270ce397fd78b5629f084f13771e5c5d1167","retrievedAt":"2026-11-18T21:16:09.474Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"d10a7c96-5856-4773-8d5c-ebfce1411d2b"},"reviewer":{"model":"gpt-6-luna","runId":"37682276-e202-44b6-8839-06dca88ec5a4"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-18T21:17:07.200Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Qin, Yiwei","url":null,"kind":"unknown"},{"name":"Xie, Muhang","url":null,"kind":"unknown"},{"name":"Li, Xuefeng","url":null,"kind":"unknown"},{"name":"Jiang, Mohan","url":null,"kind":"unknown"},{"name":"Fu, Dayuan","url":null,"kind":"unknown"},{"name":"Mi, Tiantian","url":null,"kind":"unknown"},{"name":"Guo, Qipeng","url":null,"kind":"unknown"},{"name":"Liu, Pengfei","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Qin, Yiwei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Qin, Yiwei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21315","locator":"head/meta[name=author|citation_author]"},"assertionId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T21:48:09.936Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6","sourceUrl":"https://arxiv.org/abs/2611.21315","contentSha256":"5441256af85ca1e16ca4b24bc4345ce7fcbcaf8dca36ef2c187978832695fd0c","textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observedAt":"2026-11-18T21:48:09.936Z"}},{"actor":{"name":"Xie, Muhang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Xie, Muhang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21315","locator":"head/meta[name=author|citation_author]"},"assertionId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T21:48:09.936Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6","sourceUrl":"https://arxiv.org/abs/2611.21315","contentSha256":"5441256af85ca1e16ca4b24bc4345ce7fcbcaf8dca36ef2c187978832695fd0c","textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observedAt":"2026-11-18T21:48:09.936Z"}},{"actor":{"name":"Li, Xuefeng","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Xuefeng","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21315","locator":"head/meta[name=author|citation_author]"},"assertionId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T21:48:09.936Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6","sourceUrl":"https://arxiv.org/abs/2611.21315","contentSha256":"5441256af85ca1e16ca4b24bc4345ce7fcbcaf8dca36ef2c187978832695fd0c","textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observedAt":"2026-11-18T21:48:09.936Z"}},{"actor":{"name":"Jiang, Mohan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Jiang, Mohan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21315","locator":"head/meta[name=author|citation_author]"},"assertionId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T21:48:09.936Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6","sourceUrl":"https://arxiv.org/abs/2611.21315","contentSha256":"5441256af85ca1e16ca4b24bc4345ce7fcbcaf8dca36ef2c187978832695fd0c","textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observedAt":"2026-11-18T21:48:09.936Z"}},{"actor":{"name":"Fu, Dayuan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Fu, Dayuan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21315","locator":"head/meta[name=author|citation_author]"},"assertionId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T21:48:09.936Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6","sourceUrl":"https://arxiv.org/abs/2611.21315","contentSha256":"5441256af85ca1e16ca4b24bc4345ce7fcbcaf8dca36ef2c187978832695fd0c","textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observedAt":"2026-11-18T21:48:09.936Z"}},{"actor":{"name":"Mi, Tiantian","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mi, Tiantian","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21315","locator":"head/meta[name=author|citation_author]"},"assertionId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T21:48:09.936Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6","sourceUrl":"https://arxiv.org/abs/2611.21315","contentSha256":"5441256af85ca1e16ca4b24bc4345ce7fcbcaf8dca36ef2c187978832695fd0c","textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observedAt":"2026-11-18T21:48:09.936Z"}},{"actor":{"name":"Guo, Qipeng","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Guo, Qipeng","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21315","locator":"head/meta[name=author|citation_author]"},"assertionId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T21:48:09.936Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6","sourceUrl":"https://arxiv.org/abs/2611.21315","contentSha256":"5441256af85ca1e16ca4b24bc4345ce7fcbcaf8dca36ef2c187978832695fd0c","textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observedAt":"2026-11-18T21:48:09.936Z"}},{"actor":{"name":"Liu, Pengfei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liu, Pengfei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21315","locator":"head/meta[name=author|citation_author]"},"assertionId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T21:48:09.936Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6","sourceUrl":"https://arxiv.org/abs/2611.21315","contentSha256":"5441256af85ca1e16ca4b24bc4345ce7fcbcaf8dca36ef2c187978832695fd0c","textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observedAt":"2026-11-18T21:48:09.936Z"}}],"participationCoverage":{"total":8,"returned":8,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.21315","snapshotId":"fc1077068bd8d4e0b2a579d73b6534911bdb86b0a5721752cc3a8fa4fbbcecc6","contentSha256":"5441256af85ca1e16ca4b24bc4345ce7fcbcaf8dca36ef2c187978832695fd0c","textHash":"ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7","observedAt":"2026-11-18T21:48:09.936Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/26d7816600ecc2026b64f973b52d14baf2d6a332e1d0d2e3cbada3fc238269db/source"}}}],"audits":[{"publicationId":"fdbf4c41-a9fd-45a9-810c-1feaad7e9f5f","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading correctly attributes the paper to Yiwei Qin and coauthors on arXiv cs.AI. Its definitions, findings and recommendations remain framed as the authors' claims, and the cited post-training audits are presented as audits the authors cite.","excerpt":"Authors:Yiwei Qin, Muhang Xie, Xuefeng Li, Mohan Jiang, Dayuan Fu, Tiantian Mi, Qipeng Guo, Pengfei Liu"},{"dimension":"summary","status":"uncertain","explanation":"The reading drops the paper's central thesis: benchleaking is accelerating, with two escalations documented over the past month. The title and summary present a static finding. Smallest repair: state that the authors document two escalations over the past month. The phrase 'suite-wide averages range from 1.3 ... to 16.6' could also be read as Anti-Hack scores, but these numbers are point costs. Smallest repair: say 'average cost.' Base model, pre-training, few-shot continuation, HumanEval and OpenBookQA are used without plain-language explanations or tooltips, which limits understanding for nonspecialist readers.","excerpt":"We document two escalations over the past month."},{"dimension":"date","status":"pass","explanation":"The originalPublished date of 2026-11-16 matches the arXiv v1 submission date. The feed date of 2026-11-18 was correctly not used.","excerpt":"[Submitted on 16 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"Each claim matches the abstract verbatim in substance, including the leakage percentages, the Anti-Hack definition, the 1.3–16.6 point costs and the call for decontamination reports. The question about generalization beyond the tested suite follows from the stated scope and does not assert an unsupported conclusion. The limitations are appropriate.","excerpt":"the benchmark score recomputed with every leaked example marked wrong and the number of examples unchanged"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the original arXiv abstract page. It is the most direct way to access the source.","excerpt":null}],"createdAt":"2026-11-18T21:17:07.200Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}