arXiv cs.AI: Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation

arXiv cs.AI: Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation

Authors Eve Fleisig, Joachim Baumann, Dylan Hadfield-Menell and Dirk Hovy argue that arena-style language-model rankings can be manipulated. These rankings compare anonymous models in pairs on user prompts, then use the Bradley-Terry model—a method for combining pairwise votes—to produce a ranking. The authors identify three strategies: strategic voting, including votes on irrelevant models; strategic prompting, by choosing prompts that favor a model; and strategic nomination, by inserting irrelevant models. They say these attacks can raise the rank of nearly every model they tested, with each attack moving a model up by as many as 15–20 places and attacks combined moving it up by as many as 45. The authors say they demonstrate the effects on public leaderboard data, examine the circumstances that make the rankings vulnerable, and describe safeguards for each attack. The concern is not limited to a particular leaderboard: they say the vulnerabilities also apply to preference-based evaluations using the Bradley-Terry model when prompting, voting or model submission is open. Escape Meridian interpretation: if rankings can be shifted through these mechanisms, a leaderboard position may reflect how its evaluation can be manipulated as well as how models perform. The abstract does not detail how effective the proposed safeguards are.

Publication: ada9a6f1-5205-4ee0-879f-d0cf37ce1fce

attribution — pass: The reading correctly credits the four listed authors and the arXiv cs.AI preprint. It keeps the findings attributed with phrases such as 'the authors say' and 'they say.' The broader implication is clearly labeled as Escape Meridian interpretation.

Authors:Eve Fleisig, Joachim Baumann, Dylan Hadfield-Menell, Dirk Hovy

summary — pass: The reading explains what an arena-style ranking is and what the Bradley-Terry model does in plain language. It also describes the three attack strategies, the reported rank gains, the claim of generality and the safeguards. It is understandable to a general reader. One small fidelity note: the abstract says the actors are 'unscrupulous providers' and that the issues hold for 'any' such evaluation. The draft omits who would run the attacks and softens 'any', but neither omission changes the meaning.

several manipulation strategies that unscrupulous providers could use to boost a model's rank

date — pass: The originalPublished date of 2026-11-10 matches the v1 submission date on arXiv. The later feed date of 2026-11-13 is correctly not used.

[Submitted on 10 Nov 2026]

support — pass: Each claim matches the abstract, including the rank gains of 15-20 places per attack and 45 combined, and the scope condition of open prompting, voting or model submission. Linking the numbers to public leaderboard data is a reasonable reading of the abstract. The interpretation is conditional, labeled and backed by an excerpt. The question and the limitation correctly note that the abstract does not say how effective the safeguards are.

We demonstrate these effects on public leaderboard data, analyze the circumstances that cause these vulnerabilities, and describe approaches to safeguard against each attack.

media — pass: The selected media is the verified direct link to the original arXiv abstract page, which is the most relevant way to reach the source.