arXiv cs.AI paper reports a “Prompt Lottery” in LLM-as-a-Judge evaluations

arXiv cs.AI paper reports a “Prompt Lottery” in LLM-as-a-Judge evaluations

The authors of “Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing” argue that evaluations using an LLM as a judge can change when the prompt is rephrased. They say this can happen even when the rephrasing is intended to preserve the evaluation’s meaning. They call this variation “Prompt Lottery.” Across a study covering diverse evaluation tasks, judge families, and model backbones, the authors report that rephrasing can change evaluation outcomes and, in some settings, reverse relative model rankings. In a user study with 10 human annotators, they report that judges’ agreement degraded 6–9 times more under prompt rephrasing than humans’ agreement, falling below a threshold they describe as generally accepted in usability studies. The authors argue that evaluating with a single prompt is not robust enough for ranking models. They report that aggregating judgments across multiple prompt variants improves robustness by emphasizing judgments that stay consistent. The abstract does not specify the study’s task-level results or how the aggregation performs beyond the reported improvement. How well does multi-prompt aggregation preserve reliable rankings across the evaluation tasks and judge models studied?

Publication: 7e53fff2-f4a1-44fd-8509-7230fba70d23

attribution — pass: The reading attributes the findings to the authors of the named arXiv paper and keeps them as reported claims. The title's 'arXiv cs.AI' label matches the supplied publisher, though the page lists cs.CL as the primary subject and cs.AI as a cross-list. This is not a material problem.

Authors:Ilija Subasic, Brett Levert, Andrew Rabinovich, Sajjad Pakdamansavoji, Zhao Chen

summary — pass: The summary leads with the actor and the thesis, explains LLM-as-a-Judge as using an LLM to judge evaluations, and renders 'paraphrastic rephrasings' plainly. It covers ranking reversals, the human comparison, the single-prompt conclusion and the mitigation. Two optional fidelity repairs: it drops the source's 'substantially' before 'improves robustness', and 'judges’ agreement' could say 'LLM judges’ agreement' to avoid confusion with the humans. The question is a genuine open issue.

This indicates that single-prompt evaluation is insufficiently robust for model ranking.

date — pass: The originalPublished value of 2026-10-27 matches the v1 submission date shown on the arXiv page. The later feed date is not used.

[Submitted on 27 Oct 2026]

support — pass: Each claim is directly supported by the abstract and presented as the authors' report. The limitations correctly note that the abstract-only evidence lacks task-level results and does not identify the usability threshold. No unsupported benefits, costs or real-world reliability claims are added.

we find that these variations can alter evaluation outcomes and, in some settings, reverse relative model rankings

media — pass: The selected media is the verified direct link to the original arXiv abstract page, which is the most relevant way to reach the source.