Qin, Yiwei, Xie, Muhang, Li, Xuefeng, Jiang, Mohan, Fu, Dayuan, Mi, Tiantian, Guo, Qipeng, Liu, Pengfei · Nov 16, 2026
arXiv cs.AI: Yiwei Qin and coauthors argue benchleaking is reaching pre-training
Markdown · JSON · Original source
In an paper, Yiwei Qin and coauthors call “benchleaking” the entry of a ’s own test examples into training data, where a score may then reflect recall rather than capability. They distinguish it from “benchmaxxing,” which optimizes toward a benchmark without seeing its . The authors cite audits finding explicit traces in post-training; they also report the problem has reached . In continuation tests across 20 benchmarks and four public base models, some models produced test questions and answers absent from their input. They report large differences between models, including leakage of HumanEval problems with reference implementations and OpenBookQA questions with answers in ERNIE-5.5-Flash-Base. To estimate how much scores depend on leaked examples, they define an Anti-Hack score that marks each leaked example wrong while keeping the number of examples unchanged. Their suite-wide averages range from 1.3 points for Hy3-preview-Base to 16.6 for ERNIE-5.5-Flash-Base. Because the authors describe open base models as a foundation for downstream research, they argue benchleaking reaches beyond leaderboard rankings and call for decontamination reports and contamination-adjusted scores to become standard practice.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 16, 2026 (2026-11-16).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-18T21:17:07.200Z. First observed: 2026-11-17T08:31:39.138Z. Retrieved: 2026-11-18T21:16:09.474Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The reported results cover continuation on 20 and four public base models; how consistently they generalize beyond that tested suite remains unresolved. The abstract summarizes the findings but does not provide the full methods or evidence needed here to assess how leakage was detected across the suite.
Open question
How consistently would the reported leakage and Anti-Hack score differences hold across and base models beyond the tested suite?
Source-stated
The authors define benchleaking as benchmark test examples entering training data, potentially making a score reflect recall rather than capability, and distinguish it from benchmaxxing.
We use the term benchleaking for the case in which a benchmark's own test examples enter training data, so that a score reflects recall rather than capability. This differs from benchmaxxing
Source-stated
The authors tested few-shot continuation on 20 benchmarks across four public base models and report that some models produced other test examples absent from their input.
Using few-shot continuation on 20 benchmarks across four public base models, we find that some base models write out other test examples---complete questions, options, and answers---that appear nowhere in their input.
Source-stated
The authors report that 96% of HumanEval problems and 85% of OpenBookQA questions leak in ERNIE-5.5-Flash-Base, compared with 9% and none in DeepSeek-V4-Flash-Base.
96% of HumanEval problems leak with their reference implementations and 85% of OpenBookQA questions leak with their answers, against 9% and none for DeepSeek-V4-Flash-Base.
Source-stated
The authors define an Anti-Hack score that counts leaked examples as wrong while leaving the number of examples unchanged; suite-wide average costs range from 1.3 to 16.6 points among the models cited.
Averaged over the suite, it costs DeepSeek-V4-Flash-Base 1.5 points and Hy3-preview-Base 1.3, against 5.0 for MiMo-V2.5-Base and 16.6 for ERNIE-5.5-Flash-Base.
Source-stated
The authors argue benchleaking matters beyond leaderboard rankings because open base models underpin downstream research, and call for decontamination reports and contamination-adjusted scores as standard practice.
Open base models are the foundation of much downstream research, so benchleaking now affects more than leaderboard rankings. We call for decontamination reports and contamination-adjusted scores to become standard practice.
Source and contributor context
Qin, Yiwei, Xie, Muhang, Li, Xuefeng, Jiang, Mohan, Fu, Dayuan, Mi, Tiantian, Guo, Qipeng, Liu, Pengfei · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Pre-training
A model's initial training on a large dataset, before further adaptation for particular uses.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Held-out set
Data kept aside during training and used only to test how well a model works on examples it has not seen.
Zero-shot
Doing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: fdbf4c41-a9fd-45a9-810c-1feaad7e9f5f.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T21:17:07.200Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 83f2b73c0a6f188030a7aaeb3ce1270ce397fd78b5629f084f13771e5c5d1167. Source text hash: ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7.
Revision and source-check history
Version 1 · 2026-11-18T21:17:07.200Z
arXiv cs.AI: Yiwei Qin and coauthors argue benchleaking is reaching pre-training
In an paper, Yiwei Qin and coauthors call “benchleaking” the entry of a ’s own test examples into training data, where a score may then reflect recall rather than capability. They distinguish it from “benchmaxxing,” which optimizes toward a benchmark without seeing its . The authors cite audits finding explicit traces in post-training; they also report the problem has reached . In continuation tests across 20 benchmarks and four public base models, some models produced test questions and answers absent from their input. They report large differences between models, including leakage of HumanEval problems with reference implementations and OpenBookQA questions with answers in ERNIE-5.5-Flash-Base. To estimate how much scores depend on leaked examples, they define an Anti-Hack score that marks each leaked example wrong while keeping the number of examples unchanged. Their suite-wide averages range from 1.3 points for Hy3-preview-Base to 16.6 for ERNIE-5.5-Flash-Base. Because the authors describe open base models as a foundation for downstream research, they argue benchleaking reaches beyond leaderboard rankings and call for decontamination reports and contamination-adjusted scores to become standard practice.
Original source
cs.AI · Article
Source details
Original source date: Nov 16, 2026 (2026-11-16).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-18T21:17:07.200Z. First observed: 2026-11-17T08:31:39.138Z. Retrieved: 2026-11-18T21:16:09.474Z.
Limits
The reported results cover continuation on 20 and four public base models; how consistently they generalize beyond that tested suite remains unresolved. The abstract summarizes the findings but does not provide the full methods or evidence needed here to assess how leakage was detected across the suite.
Open question
How consistently would the reported leakage and Anti-Hack score differences hold across and base models beyond the tested suite?
Source-stated
The authors define benchleaking as benchmark test examples entering training data, potentially making a score reflect recall rather than capability, and distinguish it from benchmaxxing.
We use the term benchleaking for the case in which a benchmark's own test examples enter training data, so that a score reflects recall rather than capability. This differs from benchmaxxing
Source-stated
The authors tested few-shot continuation on 20 benchmarks across four public base models and report that some models produced other test examples absent from their input.
Using few-shot continuation on 20 benchmarks across four public base models, we find that some base models write out other test examples---complete questions, options, and answers---that appear nowhere in their input.
Source-stated
The authors report that 96% of HumanEval problems and 85% of OpenBookQA questions leak in ERNIE-5.5-Flash-Base, compared with 9% and none in DeepSeek-V4-Flash-Base.
96% of HumanEval problems leak with their reference implementations and 85% of OpenBookQA questions leak with their answers, against 9% and none for DeepSeek-V4-Flash-Base.
Source-stated
The authors define an Anti-Hack score that counts leaked examples as wrong while leaving the number of examples unchanged; suite-wide average costs range from 1.3 to 16.6 points among the models cited.
Averaged over the suite, it costs DeepSeek-V4-Flash-Base 1.5 points and Hy3-preview-Base 1.3, against 5.0 for MiMo-V2.5-Base and 16.6 for ERNIE-5.5-Flash-Base.
Source-stated
The authors argue benchleaking matters beyond leaderboard rankings because open base models underpin downstream research, and call for decontamination reports and contamination-adjusted scores as standard practice.
Open base models are the foundation of much downstream research, so benchleaking now affects more than leaderboard rankings. We call for decontamination reports and contamination-adjusted scores to become standard practice.
Source and contributor context
Qin, Yiwei, Xie, Muhang, Li, Xuefeng, Jiang, Mohan, Fu, Dayuan, Mi, Tiantian, Guo, Qipeng, Liu, Pengfei · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Pre-training
A model's initial training on a large dataset, before further adaptation for particular uses.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Held-out set
Data kept aside during training and used only to test how well a model works on examples it has not seen.
Zero-shot
Doing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: fdbf4c41-a9fd-45a9-810c-1feaad7e9f5f.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T21:17:07.200Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 83f2b73c0a6f188030a7aaeb3ce1270ce397fd78b5629f084f13771e5c5d1167. Source text hash: ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7.
uncertain · 2026-11-18T21:17:07.200Z · anthropic/claude-opus-5.5
Publication: fdbf4c41-a9fd-45a9-810c-1feaad7e9f5f
attribution — pass: The reading correctly attributes the paper to Yiwei Qin and coauthors on arXiv cs.AI. Its definitions, findings and recommendations remain framed as the authors' claims, and the cited post-training audits are presented as audits the authors cite.
Authors:Yiwei Qin, Muhang Xie, Xuefeng Li, Mohan Jiang, Dayuan Fu, Tiantian Mi, Qipeng Guo, Pengfei Liu
summary — uncertain: The reading drops the paper's central thesis: benchleaking is accelerating, with two escalations documented over the past month. The title and summary present a static finding. Smallest repair: state that the authors document two escalations over the past month. The phrase 'suite-wide averages range from 1.3 ... to 16.6' could also be read as Anti-Hack scores, but these numbers are point costs. Smallest repair: say 'average cost.' Base model, pre-training, few-shot continuation, HumanEval and OpenBookQA are used without plain-language explanations or tooltips, which limits understanding for nonspecialist readers.
We document two escalations over the past month.
date — pass: The originalPublished date of 2026-11-16 matches the arXiv v1 submission date. The feed date of 2026-11-18 was correctly not used.
[Submitted on 16 Nov 2026]
support — pass: Each claim matches the abstract verbatim in substance, including the leakage percentages, the Anti-Hack definition, the 1.3–16.6 point costs and the call for decontamination reports. The question about generalization beyond the tested suite follows from the stated scope and does not assert an unsupported conclusion. The limitations are appropriate.
the benchmark score recomputed with every leaked example marked wrong and the number of examples unchanged
media — pass: The selected media is the verified direct link to the original arXiv abstract page. It is the most direct way to access the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceData kept aside during training and used only to test how well a model works on examples it has not seen.
Definition referenceA model's initial training on a large dataset, before further adaptation for particular uses.
Definition referenceDoing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA model's initial training on a large dataset, before further adaptation for particular uses.
Definition referenceDoing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceData kept aside during training and used only to test how well a model works on examples it has not seen.
Definition referenceA model's initial training on a large dataset, before further adaptation for particular uses.
Definition referenceDoing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA model's initial training on a large dataset, before further adaptation for particular uses.
Definition referenceDoing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference