# arXiv cs.AI: Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation [Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-leaderboard-hacking-preference-based-model-c288de41) Authors Eve Fleisig, Joachim Baumann, Dylan Hadfield-Menell and Dirk Hovy argue that arena-style language-model rankings can be manipulated. These rankings compare anonymous models in pairs on user prompts, then use the Bradley-Terry model—a method for combining pairwise votes—to produce a ranking. The authors identify three strategies: strategic voting, including votes on irrelevant models; strategic prompting, by choosing prompts that favor a model; and strategic nomination, by inserting irrelevant models. They say these attacks can raise the rank of nearly every model they tested, with each attack moving a model up by as many as 15–20 places and attacks combined moving it up by as many as 45. The authors say they demonstrate the effects on public leaderboard data, examine the circumstances that make the rankings vulnerable, and describe safeguards for each attack. The concern is not limited to a particular leaderboard: they say the vulnerabilities also apply to preference-based evaluations using the Bradley-Terry model when prompting, voting or model submission is open. Escape Meridian interpretation: if rankings can be shifted through these mechanisms, a leaderboard position may reflect how its evaluation can be manipulated as well as how models perform. The abstract does not detail how effective the proposed safeguards are. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Nov 10, 2026 (2026-11-10). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-13T16:18:08.557Z. First observed: 2026-11-13T07:39:54.074Z. Retrieved: 2026-11-13T16:17:01.069Z. - [Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation](https://arxiv.org/abs/2611.14769) ## Limits The evidence is the paper's abstract; it does not detail the public leaderboard data, the circumstances behind each vulnerability, or the proposed safeguards' effectiveness. ## Open question How effective are the described safeguards at limiting the ranking changes the authors report? ## Source-stated The authors identify strategic voting, strategic prompting and strategic nomination as ways to manipulate arena-style rankings. > strategic voting (individual votes that boost a model's rank, including votes on irrelevant models), strategic prompting (choosing prompts that favor a model), and strategic nomination (boosting the rank of a model by inserting irrelevant models) - [Supporting source](https://arxiv.org/abs/2611.14769) ## Source-stated On public leaderboard data, the authors report that each attack can boost a model by up to 15–20 places, and attacks combined by up to 45 places. > each attack can boost a model by up to 15-20 places in a ranking and, combined, they can boost a model by up to 45 places. - [Supporting source](https://arxiv.org/abs/2611.14769) ## Source-stated The authors say the vulnerabilities apply to preference-based evaluations using the Bradley-Terry model when prompting, voting or model submission is open. > these issues hold for any preference-based evaluation using the Bradley-Terry model that allows open prompting, voting, or model submission. - [Supporting source](https://arxiv.org/abs/2611.14769) ## Escape Meridian interpretation Escape Meridian interpretation: if ranking positions can be shifted through these attacks, a leaderboard position may reflect manipulability as well as model performance. > These manipulations boost the rank of nearly every tested model; each attack can boost a model by up to 15-20 places in a ranking - [Supporting source](https://arxiv.org/abs/2611.14769) ## Source and contributor context Fleisig, Eve, Baumann, Joachim, Hadfield-Menell, Dylan, Hovy, Dirk · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Evals Structured tests of what an AI system can do or how it behaves, run before or after release. - [Definition reference](https://developers.google.com/machine-learning/glossary#evals) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: ada9a6f1-5205-4ee0-879f-d0cf37ce1fce. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-13T16:18:08.557Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 4286629580d9f63fa60d5f62c334f61d90290428ecb02c9f7179aeff35b45a18. Source text hash: 487e0a425ea8d2a824bfb519781a2d3f9fdba50b67c4ada3d0530cf865429069. ## Revision and source-check history ### Version 1 — 2026-11-13T16:18:08.557Z Authors Eve Fleisig, Joachim Baumann, Dylan Hadfield-Menell and Dirk Hovy argue that arena-style language-model rankings can be manipulated. These rankings compare anonymous models in pairs on user prompts, then use the Bradley-Terry model—a method for combining pairwise votes—to produce a ranking. The authors identify three strategies: strategic voting, including votes on irrelevant models; strategic prompting, by choosing prompts that favor a model; and strategic nomination, by inserting irrelevant models. They say these attacks can raise the rank of nearly every model they tested, with each attack moving a model up by as many as 15–20 places and attacks combined moving it up by as many as 45. The authors say they demonstrate the effects on public leaderboard data, examine the circumstances that make the rankings vulnerable, and describe safeguards for each attack. The concern is not limited to a particular leaderboard: they say the vulnerabilities also apply to preference-based evaluations using the Bradley-Terry model when prompting, voting or model submission is open. Escape Meridian interpretation: if rankings can be shifted through these mechanisms, a leaderboard position may reflect how its evaluation can be manipulated as well as how models perform. The abstract does not detail how effective the proposed safeguards are. ## Original source arXiv cs.AI · Article Original source date: Nov 10, 2026 (2026-11-10). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-13T16:18:08.557Z. First observed: 2026-11-13T07:39:54.074Z. Retrieved: 2026-11-13T16:17:01.069Z. - [Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation](https://arxiv.org/abs/2611.14769) ## Limits The evidence is the paper's abstract; it does not detail the public leaderboard data, the circumstances behind each vulnerability, or the proposed safeguards' effectiveness. ## Open question How effective are the described safeguards at limiting the ranking changes the authors report? ## Source-stated The authors identify strategic voting, strategic prompting and strategic nomination as ways to manipulate arena-style rankings. > strategic voting (individual votes that boost a model's rank, including votes on irrelevant models), strategic prompting (choosing prompts that favor a model), and strategic nomination (boosting the rank of a model by inserting irrelevant models) - [Supporting source](https://arxiv.org/abs/2611.14769) ## Source-stated On public leaderboard data, the authors report that each attack can boost a model by up to 15–20 places, and attacks combined by up to 45 places. > each attack can boost a model by up to 15-20 places in a ranking and, combined, they can boost a model by up to 45 places. - [Supporting source](https://arxiv.org/abs/2611.14769) ## Source-stated The authors say the vulnerabilities apply to preference-based evaluations using the Bradley-Terry model when prompting, voting or model submission is open. > these issues hold for any preference-based evaluation using the Bradley-Terry model that allows open prompting, voting, or model submission. - [Supporting source](https://arxiv.org/abs/2611.14769) ## Escape Meridian interpretation Escape Meridian interpretation: if ranking positions can be shifted through these attacks, a leaderboard position may reflect manipulability as well as model performance. > These manipulations boost the rank of nearly every tested model; each attack can boost a model by up to 15-20 places in a ranking - [Supporting source](https://arxiv.org/abs/2611.14769) ## Source and contributor context Fleisig, Eve, Baumann, Joachim, Hadfield-Menell, Dylan, Hovy, Dirk · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Evals Structured tests of what an AI system can do or how it behaves, run before or after release. - [Definition reference](https://developers.google.com/machine-learning/glossary#evals) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: ada9a6f1-5205-4ee0-879f-d0cf37ce1fce. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-13T16:18:08.557Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 4286629580d9f63fa60d5f62c334f61d90290428ecb02c9f7179aeff35b45a18. Source text hash: 487e0a425ea8d2a824bfb519781a2d3f9fdba50b67c4ada3d0530cf865429069. ### passed — 2026-11-13T16:18:08.557Z — anthropic/claude-opus-5.5 Publication: ada9a6f1-5205-4ee0-879f-d0cf37ce1fce attribution — pass: The reading correctly credits the four listed authors and the arXiv cs.AI preprint. It keeps the findings attributed with phrases such as 'the authors say' and 'they say.' The broader implication is clearly labeled as Escape Meridian interpretation. > Authors:Eve Fleisig, Joachim Baumann, Dylan Hadfield-Menell, Dirk Hovy summary — pass: The reading explains what an arena-style ranking is and what the Bradley-Terry model does in plain language. It also describes the three attack strategies, the reported rank gains, the claim of generality and the safeguards. It is understandable to a general reader. One small fidelity note: the abstract says the actors are 'unscrupulous providers' and that the issues hold for 'any' such evaluation. The draft omits who would run the attacks and softens 'any', but neither omission changes the meaning. > several manipulation strategies that unscrupulous providers could use to boost a model's rank date — pass: The originalPublished date of 2026-11-10 matches the v1 submission date on arXiv. The later feed date of 2026-11-13 is correctly not used. > \[Submitted on 10 Nov 2026\] support — pass: Each claim matches the abstract, including the rank gains of 15-20 places per attack and 45 combined, and the scope condition of open prompting, voting or model submission. Linking the numbers to public leaderboard data is a reasonable reading of the abstract. The interpretation is conditional, labeled and backed by an excerpt. The question and the limitation correctly note that the abstract does not say how effective the safeguards are. > We demonstrate these effects on public leaderboard data, analyze the circumstances that cause these vulnerabilities, and describe approaches to safeguard against each attack. media — pass: The selected media is the verified direct link to the original arXiv abstract page, which is the most relevant way to reach the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/c288de419a80fbea93bc6d54a25a4edf0a8c4c11da59c54b72a1c1e6739caf7d/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-leaderboard-hacking-preference-based-model-c288de41.json)