# arXiv cs.AI paper reports a “Prompt Lottery” in LLM-as-a-Judge evaluations [Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-paper-reports-a-prompt-lottery-in-llm-as-a-judge-30ec9bd5) The authors of “Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing” argue that evaluations using an LLM as a judge can change when the prompt is rephrased. They say this can happen even when the rephrasing is intended to preserve the evaluation’s meaning. They call this variation “Prompt Lottery.” Across a study covering diverse evaluation tasks, judge families, and model backbones, the authors report that rephrasing can change evaluation outcomes and, in some settings, reverse relative model rankings. In a user study with 10 human annotators, they report that judges’ agreement degraded 6–9 times more under prompt rephrasing than humans’ agreement, falling below a threshold they describe as generally accepted in usability studies. The authors argue that evaluating with a single prompt is not robust enough for ranking models. They report that aggregating judgments across multiple prompt variants improves robustness by emphasizing judgments that stay consistent. The abstract does not specify the study’s task-level results or how the aggregation performs beyond the reported improvement. How well does multi-prompt aggregation preserve reliable rankings across the evaluation tasks and judge models studied? Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Oct 27, 2026 (2026-10-27). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-11T08:26:17.534Z. First observed: 2026-10-30T06:45:29.976Z. Retrieved: 2026-10-31T05:18:33.876Z. - [Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing](https://arxiv.org/abs/2610.38076) ## Limits The evidence provided is the paper abstract, not its full methods or results; it does not give task-level findings or details needed to assess how the reported aggregation performs across settings. The abstract does not identify the usability threshold or provide further detail about the human annotation study. ## Open question How well does multi-prompt aggregation preserve reliable rankings across the evaluation tasks and judge models studied? ## Source-stated The authors introduce “Prompt Lottery” to describe differing LLM-as-a-Judge judgments under paraphrastic rephrasings intended to preserve evaluation semantics. > We introduce \\textit{Prompt Lottery}, a phenomenon in which paraphrastic rephrasings produce different LLMaJ judgments. - [Supporting source](https://arxiv.org/abs/2610.38076) ## Source-stated The authors report that prompt rephrasing can change evaluation outcomes and sometimes reverse relative model rankings across their study. > these variations can alter evaluation outcomes and, in some settings, reverse relative model rankings. - [Supporting source](https://arxiv.org/abs/2610.38076) ## Source-stated In a user study with 10 human annotators, the authors report that judges’ agreement degraded 6–9 times more than humans’ under prompt rephrasings, falling below a threshold described as generally accepted in usability studies. > A user study with 10 human annotators shows that LLMaJ judges exhibit 6-9$\\times$ larger agreement degradation under prompt rephrasings than humans, falling below the generally accepted threshold in usability studies. - [Supporting source](https://arxiv.org/abs/2610.38076) ## Source-stated The authors say multi-prompt aggregation improves robustness by emphasizing judgments consistent across prompt variants. > Finally, we show that multi-prompt aggregation substantially improves robustness by emphasizing judgments that remain consistent across prompt variants. - [Supporting source](https://arxiv.org/abs/2610.38076) ## Source and contributor context Subasic, Ilija, Levert, Brett, Rabinovich, Andrew, Pakdamansavoji, Sajjad, Chen, Zhao · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 7e53fff2-f4a1-44fd-8509-7230fba70d23. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-11T08:26:17.534Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: af7bd3e3876224dca629bae81bd40e2fa6208218a5c4a30943e3717cf013f6e5. Source text hash: 60f0a5492e3d5273f762a65904d086e186fbac666b7885749d54b30d450c2d5c. ## Revision and source-check history ### Version 1 — 2026-11-11T08:26:17.534Z The authors of “Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing” argue that evaluations using an LLM as a judge can change when the prompt is rephrased. They say this can happen even when the rephrasing is intended to preserve the evaluation’s meaning. They call this variation “Prompt Lottery.” Across a study covering diverse evaluation tasks, judge families, and model backbones, the authors report that rephrasing can change evaluation outcomes and, in some settings, reverse relative model rankings. In a user study with 10 human annotators, they report that judges’ agreement degraded 6–9 times more under prompt rephrasing than humans’ agreement, falling below a threshold they describe as generally accepted in usability studies. The authors argue that evaluating with a single prompt is not robust enough for ranking models. They report that aggregating judgments across multiple prompt variants improves robustness by emphasizing judgments that stay consistent. The abstract does not specify the study’s task-level results or how the aggregation performs beyond the reported improvement. How well does multi-prompt aggregation preserve reliable rankings across the evaluation tasks and judge models studied? ## Original source arXiv cs.AI · Article Original source date: Oct 27, 2026 (2026-10-27). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-11T08:26:17.534Z. First observed: 2026-10-30T06:45:29.976Z. Retrieved: 2026-10-31T05:18:33.876Z. - [Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing](https://arxiv.org/abs/2610.38076) ## Limits The evidence provided is the paper abstract, not its full methods or results; it does not give task-level findings or details needed to assess how the reported aggregation performs across settings. The abstract does not identify the usability threshold or provide further detail about the human annotation study. ## Open question How well does multi-prompt aggregation preserve reliable rankings across the evaluation tasks and judge models studied? ## Source-stated The authors introduce “Prompt Lottery” to describe differing LLM-as-a-Judge judgments under paraphrastic rephrasings intended to preserve evaluation semantics. > We introduce \\textit{Prompt Lottery}, a phenomenon in which paraphrastic rephrasings produce different LLMaJ judgments. - [Supporting source](https://arxiv.org/abs/2610.38076) ## Source-stated The authors report that prompt rephrasing can change evaluation outcomes and sometimes reverse relative model rankings across their study. > these variations can alter evaluation outcomes and, in some settings, reverse relative model rankings. - [Supporting source](https://arxiv.org/abs/2610.38076) ## Source-stated In a user study with 10 human annotators, the authors report that judges’ agreement degraded 6–9 times more than humans’ under prompt rephrasings, falling below a threshold described as generally accepted in usability studies. > A user study with 10 human annotators shows that LLMaJ judges exhibit 6-9$\\times$ larger agreement degradation under prompt rephrasings than humans, falling below the generally accepted threshold in usability studies. - [Supporting source](https://arxiv.org/abs/2610.38076) ## Source-stated The authors say multi-prompt aggregation improves robustness by emphasizing judgments consistent across prompt variants. > Finally, we show that multi-prompt aggregation substantially improves robustness by emphasizing judgments that remain consistent across prompt variants. - [Supporting source](https://arxiv.org/abs/2610.38076) ## Source and contributor context Subasic, Ilija, Levert, Brett, Rabinovich, Andrew, Pakdamansavoji, Sajjad, Chen, Zhao · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 7e53fff2-f4a1-44fd-8509-7230fba70d23. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-11T08:26:17.534Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: af7bd3e3876224dca629bae81bd40e2fa6208218a5c4a30943e3717cf013f6e5. Source text hash: 60f0a5492e3d5273f762a65904d086e186fbac666b7885749d54b30d450c2d5c. ### passed — 2026-11-11T08:26:17.534Z — anthropic/claude-opus-5.5 Publication: 7e53fff2-f4a1-44fd-8509-7230fba70d23 attribution — pass: The reading attributes the findings to the authors of the named arXiv paper and keeps them as reported claims. The title's 'arXiv cs.AI' label matches the supplied publisher, though the page lists cs.CL as the primary subject and cs.AI as a cross-list. This is not a material problem. > Authors:Ilija Subasic, Brett Levert, Andrew Rabinovich, Sajjad Pakdamansavoji, Zhao Chen summary — pass: The summary leads with the actor and the thesis, explains LLM-as-a-Judge as using an LLM to judge evaluations, and renders 'paraphrastic rephrasings' plainly. It covers ranking reversals, the human comparison, the single-prompt conclusion and the mitigation. Two optional fidelity repairs: it drops the source's 'substantially' before 'improves robustness', and 'judges’ agreement' could say 'LLM judges’ agreement' to avoid confusion with the humans. The question is a genuine open issue. > This indicates that single-prompt evaluation is insufficiently robust for model ranking. date — pass: The originalPublished value of 2026-10-27 matches the v1 submission date shown on the arXiv page. The later feed date is not used. > \[Submitted on 27 Oct 2026\] support — pass: Each claim is directly supported by the abstract and presented as the authors' report. The limitations correctly note that the abstract-only evidence lacks task-level results and does not identify the usability threshold. No unsupported benefits, costs or real-world reliability claims are added. > we find that these variations can alter evaluation outcomes and, in some settings, reverse relative model rankings media — pass: The selected media is the verified direct link to the original arXiv abstract page, which is the most relevant way to reach the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/30ec9bd5c0549dac85b4e407f11c4492cadaa385e0b953d53f4b7be82f9d76b7/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-paper-reports-a-prompt-lottery-in-llm-as-a-judge-30ec9bd5.json)