# arXiv cs.AI: Yiwei Qin and coauthors argue benchleaking is reaching pre-training

[Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-yiwei-qin-and-coauthors-argue-benchleaking-is-26d78166)

In an arXiv paper, Yiwei Qin and coauthors call “benchleaking” the entry of a benchmark’s own test examples into training data, where a score may then reflect recall rather than capability. They distinguish it from “benchmaxxing,” which optimizes toward a benchmark without seeing its test set. The authors cite audits finding explicit traces in post-training; they also report the problem has reached pre-training. In few-shot continuation tests across 20 benchmarks and four public base models, some models produced test questions and answers absent from their input. They report large differences between models, including leakage of HumanEval problems with reference implementations and OpenBookQA questions with answers in ERNIE-5.5-Flash-Base. To estimate how much scores depend on leaked examples, they define an Anti-Hack score that marks each leaked example wrong while keeping the number of examples unchanged. Their suite-wide averages range from 1.3 points for Hy3-preview-Base to 16.6 for ERNIE-5.5-Flash-Base. Because the authors describe open base models as a foundation for downstream research, they argue benchleaking reaches beyond leaderboard rankings and call for decontamination reports and contamination-adjusted scores to become standard practice.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv cs.AI · Article

Original source date: Nov 16, 2026 (2026-11-16).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-18T21:17:07.200Z. First observed: 2026-11-17T08:31:39.138Z. Retrieved: 2026-11-18T21:16:09.474Z.

- [Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models](https://arxiv.org/abs/2611.21315)

## Limits

The reported pre-training results cover few-shot continuation on 20 benchmarks and four public base models; how consistently they generalize beyond that tested suite remains unresolved. The abstract summarizes the findings but does not provide the full methods or evidence needed here to assess how leakage was detected across the suite.


## Open question

How consistently would the reported leakage and Anti-Hack score differences hold across benchmarks and base models beyond the tested suite?


## Source-stated

The authors define benchleaking as benchmark test examples entering training data, potentially making a score reflect recall rather than capability, and distinguish it from benchmaxxing.

> We use the term benchleaking for the case in which a benchmark's own test examples enter training data, so that a score reflects recall rather than capability. This differs from benchmaxxing

- [Supporting source](https://arxiv.org/abs/2611.21315)

## Source-stated

The authors tested few-shot continuation on 20 benchmarks across four public base models and report that some models produced other test examples absent from their input.

> Using few-shot continuation on 20 benchmarks across four public base models, we find that some base models write out other test examples---complete questions, options, and answers---that appear nowhere in their input.

- [Supporting source](https://arxiv.org/abs/2611.21315)

## Source-stated

The authors report that 96% of HumanEval problems and 85% of OpenBookQA questions leak in ERNIE-5.5-Flash-Base, compared with 9% and none in DeepSeek-V4-Flash-Base.

> 96% of HumanEval problems leak with their reference implementations and 85% of OpenBookQA questions leak with their answers, against 9% and none for DeepSeek-V4-Flash-Base.

- [Supporting source](https://arxiv.org/abs/2611.21315)

## Source-stated

The authors define an Anti-Hack score that counts leaked examples as wrong while leaving the number of examples unchanged; suite-wide average costs range from 1.3 to 16.6 points among the models cited.

> Averaged over the suite, it costs DeepSeek-V4-Flash-Base 1.5 points and Hy3-preview-Base 1.3, against 5.0 for MiMo-V2.5-Base and 16.6 for ERNIE-5.5-Flash-Base.

- [Supporting source](https://arxiv.org/abs/2611.21315)

## Source-stated

The authors argue benchleaking matters beyond leaderboard rankings because open base models underpin downstream research, and call for decontamination reports and contamination-adjusted scores as standard practice.

> Open base models are the foundation of much downstream research, so benchleaking now affects more than leaderboard rankings. We call for decontamination reports and contamination-adjusted scores to become standard practice.

- [Supporting source](https://arxiv.org/abs/2611.21315)

## Source and contributor context

Qin, Yiwei, Xie, Muhang, Li, Xuefeng, Jiang, Mohan, Fu, Dayuan, Mi, Tiantian, Guo, Qipeng, Liu, Pengfei · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Pre-training

A model's initial training on a large dataset, before further adaptation for particular uses.

- [Definition reference](https://developers.google.com/machine-learning/glossary#pre-training)

## Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

- [Definition reference](https://arxiv.org/abs/2211.09110)

## Held-out set

Data kept aside during training and used only to test how well a model works on examples it has not seen.

- [Definition reference](https://developers.google.com/machine-learning/glossary#holdout-data)

## Zero-shot

Doing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).

- [Definition reference](https://developers.google.com/machine-learning/glossary#zero-shot-prompting)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: fdbf4c41-a9fd-45a9-810c-1feaad7e9f5f.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T21:17:07.200Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 83f2b73c0a6f188030a7aaeb3ce1270ce397fd78b5629f084f13771e5c5d1167. Source text hash: ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7.


## Revision and source-check history

### Version 1 — 2026-11-18T21:17:07.200Z

In an arXiv paper, Yiwei Qin and coauthors call “benchleaking” the entry of a benchmark’s own test examples into training data, where a score may then reflect recall rather than capability. They distinguish it from “benchmaxxing,” which optimizes toward a benchmark without seeing its test set. The authors cite audits finding explicit traces in post-training; they also report the problem has reached pre-training. In few-shot continuation tests across 20 benchmarks and four public base models, some models produced test questions and answers absent from their input. They report large differences between models, including leakage of HumanEval problems with reference implementations and OpenBookQA questions with answers in ERNIE-5.5-Flash-Base. To estimate how much scores depend on leaked examples, they define an Anti-Hack score that marks each leaked example wrong while keeping the number of examples unchanged. Their suite-wide averages range from 1.3 points for Hy3-preview-Base to 16.6 for ERNIE-5.5-Flash-Base. Because the authors describe open base models as a foundation for downstream research, they argue benchleaking reaches beyond leaderboard rankings and call for decontamination reports and contamination-adjusted scores to become standard practice.

## Original source

arXiv cs.AI · Article

Original source date: Nov 16, 2026 (2026-11-16).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-18T21:17:07.200Z. First observed: 2026-11-17T08:31:39.138Z. Retrieved: 2026-11-18T21:16:09.474Z.

- [Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models](https://arxiv.org/abs/2611.21315)

## Limits

The reported pre-training results cover few-shot continuation on 20 benchmarks and four public base models; how consistently they generalize beyond that tested suite remains unresolved. The abstract summarizes the findings but does not provide the full methods or evidence needed here to assess how leakage was detected across the suite.


## Open question

How consistently would the reported leakage and Anti-Hack score differences hold across benchmarks and base models beyond the tested suite?


## Source-stated

The authors define benchleaking as benchmark test examples entering training data, potentially making a score reflect recall rather than capability, and distinguish it from benchmaxxing.

> We use the term benchleaking for the case in which a benchmark's own test examples enter training data, so that a score reflects recall rather than capability. This differs from benchmaxxing

- [Supporting source](https://arxiv.org/abs/2611.21315)

## Source-stated

The authors tested few-shot continuation on 20 benchmarks across four public base models and report that some models produced other test examples absent from their input.

> Using few-shot continuation on 20 benchmarks across four public base models, we find that some base models write out other test examples---complete questions, options, and answers---that appear nowhere in their input.

- [Supporting source](https://arxiv.org/abs/2611.21315)

## Source-stated

The authors report that 96% of HumanEval problems and 85% of OpenBookQA questions leak in ERNIE-5.5-Flash-Base, compared with 9% and none in DeepSeek-V4-Flash-Base.

> 96% of HumanEval problems leak with their reference implementations and 85% of OpenBookQA questions leak with their answers, against 9% and none for DeepSeek-V4-Flash-Base.

- [Supporting source](https://arxiv.org/abs/2611.21315)

## Source-stated

The authors define an Anti-Hack score that counts leaked examples as wrong while leaving the number of examples unchanged; suite-wide average costs range from 1.3 to 16.6 points among the models cited.

> Averaged over the suite, it costs DeepSeek-V4-Flash-Base 1.5 points and Hy3-preview-Base 1.3, against 5.0 for MiMo-V2.5-Base and 16.6 for ERNIE-5.5-Flash-Base.

- [Supporting source](https://arxiv.org/abs/2611.21315)

## Source-stated

The authors argue benchleaking matters beyond leaderboard rankings because open base models underpin downstream research, and call for decontamination reports and contamination-adjusted scores as standard practice.

> Open base models are the foundation of much downstream research, so benchleaking now affects more than leaderboard rankings. We call for decontamination reports and contamination-adjusted scores to become standard practice.

- [Supporting source](https://arxiv.org/abs/2611.21315)

## Source and contributor context

Qin, Yiwei, Xie, Muhang, Li, Xuefeng, Jiang, Mohan, Fu, Dayuan, Mi, Tiantian, Guo, Qipeng, Liu, Pengfei · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Pre-training

A model's initial training on a large dataset, before further adaptation for particular uses.

- [Definition reference](https://developers.google.com/machine-learning/glossary#pre-training)

## Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

- [Definition reference](https://arxiv.org/abs/2211.09110)

## Held-out set

Data kept aside during training and used only to test how well a model works on examples it has not seen.

- [Definition reference](https://developers.google.com/machine-learning/glossary#holdout-data)

## Zero-shot

Doing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).

- [Definition reference](https://developers.google.com/machine-learning/glossary#zero-shot-prompting)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: fdbf4c41-a9fd-45a9-810c-1feaad7e9f5f.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T21:17:07.200Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 83f2b73c0a6f188030a7aaeb3ce1270ce397fd78b5629f084f13771e5c5d1167. Source text hash: ecdb68be93a5ff879044e2402eed5bebe85d1c2d9c0474aaf77b8b9f32b497c7.


### uncertain — 2026-11-18T21:17:07.200Z — anthropic/claude-opus-5.5

Publication: fdbf4c41-a9fd-45a9-810c-1feaad7e9f5f

attribution — pass: The reading correctly attributes the paper to Yiwei Qin and coauthors on arXiv cs.AI. Its definitions, findings and recommendations remain framed as the authors' claims, and the cited post-training audits are presented as audits the authors cite.

> Authors:Yiwei Qin, Muhang Xie, Xuefeng Li, Mohan Jiang, Dayuan Fu, Tiantian Mi, Qipeng Guo, Pengfei Liu

summary — uncertain: The reading drops the paper's central thesis: benchleaking is accelerating, with two escalations documented over the past month. The title and summary present a static finding. Smallest repair: state that the authors document two escalations over the past month. The phrase 'suite-wide averages range from 1.3 ... to 16.6' could also be read as Anti-Hack scores, but these numbers are point costs. Smallest repair: say 'average cost.' Base model, pre-training, few-shot continuation, HumanEval and OpenBookQA are used without plain-language explanations or tooltips, which limits understanding for nonspecialist readers.

> We document two escalations over the past month.

date — pass: The originalPublished date of 2026-11-16 matches the arXiv v1 submission date. The feed date of 2026-11-18 was correctly not used.

> \[Submitted on 16 Nov 2026\]

support — pass: Each claim matches the abstract verbatim in substance, including the leakage percentages, the Anti-Hack definition, the 1.3–16.6 point costs and the call for decontamination reports. The question about generalization beyond the tested suite follows from the stated scope and does not assert an unsupported conclusion. The limitations are appropriate.

> the benchmark score recomputed with every leaked example marked wrong and the number of examples unchanged

media — pass: The selected media is the verified direct link to the original arXiv abstract page. It is the most direct way to access the source.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/26d7816600ecc2026b64f973b52d14baf2d6a332e1d0d2e3cbada3fc238269db/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-yiwei-qin-and-coauthors-argue-benchleaking-is-26d78166.json)
