{"id":"30ec9bd5c0549dac85b4e407f11c4492cadaa385e0b953d53f4b7be82f9d76b7","title":"arXiv cs.AI paper reports a “Prompt Lottery” in LLM-as-a-Judge evaluations","argument":"The authors of “Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing” argue that evaluations using an LLM as a judge can change when the prompt is rephrased. They say this can happen even when the rephrasing is intended to preserve the evaluation’s meaning. They call this variation “Prompt Lottery.”\n\nAcross a study covering diverse evaluation tasks, judge families, and model backbones, the authors report that rephrasing can change evaluation outcomes and, in some settings, reverse relative model rankings. In a user study with 10 human annotators, they report that judges’ agreement degraded 6–9 times more under prompt rephrasing than humans’ agreement, falling below a threshold they describe as generally accepted in usability studies.\n\nThe authors argue that evaluating with a single prompt is not robust enough for ranking models. They report that aggregating judgments across multiple prompt variants improves robustness by emphasizing judgments that stay consistent. The abstract does not specify the study’s task-level results or how the aggregation performs beyond the reported improvement.\n\nHow well does multi-prompt aggregation preserve reliable rankings across the evaluation tasks and judge models studied?","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence provided is the paper abstract, not its full methods or results; it does not give task-level findings or details needed to assess how the reported aggregation performs across settings. The abstract does not identify the usability threshold or provide further detail about the human annotation study.","speaker":{"name":"Subasic, Ilija, Levert, Brett, Rabinovich, Andrew, Pakdamansavoji, Sajjad, Chen, Zhao","roleAtSource":"Article author"},"source":{"title":"Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.38076","publishedAt":"2026-10-27","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce “Prompt Lottery” to describe differing LLM-as-a-Judge judgments under paraphrastic rephrasings intended to preserve evaluation semantics.","excerpt":"We introduce \\textit{Prompt Lottery}, a phenomenon in which paraphrastic rephrasings produce different LLMaJ judgments.","url":"https://arxiv.org/abs/2610.38076","relationship":"Source-stated"},{"text":"The authors report that prompt rephrasing can change evaluation outcomes and sometimes reverse relative model rankings across their study.","excerpt":"these variations can alter evaluation outcomes and, in some settings, reverse relative model rankings.","url":"https://arxiv.org/abs/2610.38076","relationship":"Source-stated"},{"text":"In a user study with 10 human annotators, the authors report that judges’ agreement degraded 6–9 times more than humans’ under prompt rephrasings, falling below a threshold described as generally accepted in usability studies.","excerpt":"A user study with 10 human annotators shows that LLMaJ judges exhibit 6-9$\\times$ larger agreement degradation under prompt rephrasings than humans, falling below the generally accepted threshold in usability studies.","url":"https://arxiv.org/abs/2610.38076","relationship":"Source-stated"},{"text":"The authors say multi-prompt aggregation improves robustness by emphasizing judgments consistent across prompt variants.","excerpt":"Finally, we show that multi-prompt aggregation substantially improves robustness by emphasizing judgments that remain consistent across prompt variants.","url":"https://arxiv.org/abs/2610.38076","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.38076","checkedAt":"2026-11-12T01:30:20.513Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well does multi-prompt aggregation preserve reliable rankings across the evaluation tasks and judge models studied?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI paper reports a “Prompt Lottery” in LLM-as-a-Judge evaluations","summary":"The authors of “Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing” argue that evaluations using an LLM as a judge can change when the prompt is rephrased. They say this can happen even when the rephrasing is intended to preserve the evaluation’s meaning. They call this variation “Prompt Lottery.”\n\nAcross a study covering diverse evaluation tasks, judge families, and model backbones, the authors report that rephrasing can change evaluation outcomes and, in some settings, reverse relative model rankings. In a user study with 10 human annotators, they report that judges’ agreement degraded 6–9 times more under prompt rephrasing than humans’ agreement, falling below a threshold they describe as generally accepted in usability studies.\n\nThe authors argue that evaluating with a single prompt is not robust enough for ranking models. They report that aggregating judgments across multiple prompt variants improves robustness by emphasizing judgments that stay consistent. The abstract does not specify the study’s task-level results or how the aggregation performs beyond the reported improvement.\n\nHow well does multi-prompt aggregation preserve reliable rankings across the evaluation tasks and judge models studied?","question":"How well does multi-prompt aggregation preserve reliable rankings across the evaluation tasks and judge models studied?","claims":[{"text":"The authors introduce “Prompt Lottery” to describe differing LLM-as-a-Judge judgments under paraphrastic rephrasings intended to preserve evaluation semantics.","excerpt":"We introduce \\textit{Prompt Lottery}, a phenomenon in which paraphrastic rephrasings produce different LLMaJ judgments.","support":"source-stated"},{"text":"The authors report that prompt rephrasing can change evaluation outcomes and sometimes reverse relative model rankings across their study.","excerpt":"these variations can alter evaluation outcomes and, in some settings, reverse relative model rankings.","support":"source-stated"},{"text":"In a user study with 10 human annotators, the authors report that judges’ agreement degraded 6–9 times more than humans’ under prompt rephrasings, falling below a threshold described as generally accepted in usability studies.","excerpt":"A user study with 10 human annotators shows that LLMaJ judges exhibit 6-9$\\times$ larger agreement degradation under prompt rephrasings than humans, falling below the generally accepted threshold in usability studies.","support":"source-stated"},{"text":"The authors say multi-prompt aggregation improves robustness by emphasizing judgments consistent across prompt variants.","excerpt":"Finally, we show that multi-prompt aggregation substantially improves robustness by emphasizing judgments that remain consistent across prompt variants.","support":"source-stated"}],"limitations":["The evidence provided is the paper abstract, not its full methods or results; it does not give task-level findings or details needed to assess how the reported aggregation performs across settings.","The abstract does not identify the usability threshold or provide further detail about the human annotation study."],"mediaId":"9e3d9742e5df55c721f6e0dc72086997d2c956a59055fc2df3bd06ddc09dbbc9","correctionReason":null,"id":"30ec9bd5c0549dac85b4e407f11c4492cadaa385e0b953d53f4b7be82f9d76b7","publicationId":"7e53fff2-f4a1-44fd-8509-7230fba70d23","version":1,"sourceUrl":"https://arxiv.org/abs/2610.38076","sourceTitle":"Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-27","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-30T06:45:29.976Z","createdAt":"2026-11-11T08:26:17.534Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"9e3d9742e5df55c721f6e0dc72086997d2c956a59055fc2df3bd06ddc09dbbc9","kind":"article","url":"https://arxiv.org/abs/2610.38076","sourceUrl":"https://arxiv.org/abs/2610.38076","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-31T05:18:34.115Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.38076","checkedAt":"2026-11-12T01:30:20.513Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"9e3d9742e5df55c721f6e0dc72086997d2c956a59055fc2df3bd06ddc09dbbc9","confidence":0.81,"provider":"typesafe","model":"jev-1.13.0","inputHash":"95a71cd269687241ecf7643c83a978e9d946e85834b49ca061c109852725fff1","decidedAt":"2026-10-31T05:18:35.826Z"},"provenance":{"textHash":"60f0a5492e3d5273f762a65904d086e186fbac666b7885749d54b30d450c2d5c","observationId":"af7bd3e3876224dca629bae81bd40e2fa6208218a5c4a30943e3717cf013f6e5","retrievedAt":"2026-10-31T05:18:33.876Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"68efd398-3c4d-431f-868c-9b2a3f00cc5f"},"reviewer":{"model":"gpt-6-luna","runId":"fdcc0b69-7891-45c6-80e1-240fe7541e7a"}},"audit":{"status":"passed","reviewedAt":"2026-11-11T08:26:17.534Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Subasic, Ilija","url":null,"kind":"unknown"},{"name":"Levert, Brett","url":null,"kind":"unknown"},{"name":"Rabinovich, Andrew","url":null,"kind":"unknown"},{"name":"Pakdamansavoji, Sajjad","url":null,"kind":"unknown"},{"name":"Chen, Zhao","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Subasic, Ilija","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Subasic, Ilija","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38076","locator":"head/meta[name=author|citation_author]"},"assertionId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-31T05:18:33.876Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b","sourceUrl":"https://arxiv.org/abs/2610.38076","contentSha256":"f0890cc497428439e7555ac774d25755d3e9cdcbc6fd27d86f08bf2c2b1b15e7","textHash":"60f0a5492e3d5273f762a65904d086e186fbac666b7885749d54b30d450c2d5c","observedAt":"2026-10-31T05:18:33.876Z"}},{"actor":{"name":"Levert, Brett","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Levert, Brett","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38076","locator":"head/meta[name=author|citation_author]"},"assertionId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-31T05:18:33.876Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b","sourceUrl":"https://arxiv.org/abs/2610.38076","contentSha256":"f0890cc497428439e7555ac774d25755d3e9cdcbc6fd27d86f08bf2c2b1b15e7","textHash":"60f0a5492e3d5273f762a65904d086e186fbac666b7885749d54b30d450c2d5c","observedAt":"2026-10-31T05:18:33.876Z"}},{"actor":{"name":"Rabinovich, Andrew","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Rabinovich, Andrew","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38076","locator":"head/meta[name=author|citation_author]"},"assertionId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-31T05:18:33.876Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b","sourceUrl":"https://arxiv.org/abs/2610.38076","contentSha256":"f0890cc497428439e7555ac774d25755d3e9cdcbc6fd27d86f08bf2c2b1b15e7","textHash":"60f0a5492e3d5273f762a65904d086e186fbac666b7885749d54b30d450c2d5c","observedAt":"2026-10-31T05:18:33.876Z"}},{"actor":{"name":"Pakdamansavoji, Sajjad","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Pakdamansavoji, Sajjad","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38076","locator":"head/meta[name=author|citation_author]"},"assertionId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-31T05:18:33.876Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b","sourceUrl":"https://arxiv.org/abs/2610.38076","contentSha256":"f0890cc497428439e7555ac774d25755d3e9cdcbc6fd27d86f08bf2c2b1b15e7","textHash":"60f0a5492e3d5273f762a65904d086e186fbac666b7885749d54b30d450c2d5c","observedAt":"2026-10-31T05:18:33.876Z"}},{"actor":{"name":"Chen, Zhao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chen, Zhao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38076","locator":"head/meta[name=author|citation_author]"},"assertionId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-31T05:18:33.876Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b","sourceUrl":"https://arxiv.org/abs/2610.38076","contentSha256":"f0890cc497428439e7555ac774d25755d3e9cdcbc6fd27d86f08bf2c2b1b15e7","textHash":"60f0a5492e3d5273f762a65904d086e186fbac666b7885749d54b30d450c2d5c","observedAt":"2026-10-31T05:18:33.876Z"}}],"participationCoverage":{"total":5,"returned":5,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.38076","snapshotId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b","contentSha256":"f0890cc497428439e7555ac774d25755d3e9cdcbc6fd27d86f08bf2c2b1b15e7","textHash":"60f0a5492e3d5273f762a65904d086e186fbac666b7885749d54b30d450c2d5c","observedAt":"2026-10-31T05:18:33.876Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/30ec9bd5c0549dac85b4e407f11c4492cadaa385e0b953d53f4b7be82f9d76b7/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-paper-reports-a-prompt-lottery-in-llm-as-a-judge-30ec9bd5","formats":{"html":"/readings/arxiv-cs-ai-paper-reports-a-prompt-lottery-in-llm-as-a-judge-30ec9bd5","markdown":"/readings/arxiv-cs-ai-paper-reports-a-prompt-lottery-in-llm-as-a-judge-30ec9bd5.md","json":"/readings/arxiv-cs-ai-paper-reports-a-prompt-lottery-in-llm-as-a-judge-30ec9bd5.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"30ec9bd5c0549dac85b4e407f11c4492cadaa385e0b953d53f4b7be82f9d76b7","revision":"7e53fff2-f4a1-44fd-8509-7230fba70d23","revisionKind":"publication-id","title":"arXiv cs.AI paper reports a “Prompt Lottery” in LLM-as-a-Judge evaluations","url":"https://escapemeridian.com/readings/arxiv-cs-ai-paper-reports-a-prompt-lottery-in-llm-as-a-judge-30ec9bd5"},"source":{"type":"source","id":"https://arxiv.org/abs/2610.38076","revision":"60f0a5492e3d5273f762a65904d086e186fbac666b7885749d54b30d450c2d5c","revisionKind":"sha256-source-text","title":"Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing","url":"https://arxiv.org/abs/2610.38076"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"30ec9bd5c0549dac85b4e407f11c4492cadaa385e0b953d53f4b7be82f9d76b7","title":"arXiv cs.AI paper reports a “Prompt Lottery” in LLM-as-a-Judge evaluations","argument":"The authors of “Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing” argue that evaluations using an LLM as a judge can change when the prompt is rephrased. They say this can happen even when the rephrasing is intended to preserve the evaluation’s meaning. They call this variation “Prompt Lottery.”\n\nAcross a study covering diverse evaluation tasks, judge families, and model backbones, the authors report that rephrasing can change evaluation outcomes and, in some settings, reverse relative model rankings. In a user study with 10 human annotators, they report that judges’ agreement degraded 6–9 times more under prompt rephrasing than humans’ agreement, falling below a threshold they describe as generally accepted in usability studies.\n\nThe authors argue that evaluating with a single prompt is not robust enough for ranking models. They report that aggregating judgments across multiple prompt variants improves robustness by emphasizing judgments that stay consistent. The abstract does not specify the study’s task-level results or how the aggregation performs beyond the reported improvement.\n\nHow well does multi-prompt aggregation preserve reliable rankings across the evaluation tasks and judge models studied?","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence provided is the paper abstract, not its full methods or results; it does not give task-level findings or details needed to assess how the reported aggregation performs across settings. The abstract does not identify the usability threshold or provide further detail about the human annotation study.","speaker":{"name":"Subasic, Ilija, Levert, Brett, Rabinovich, Andrew, Pakdamansavoji, Sajjad, Chen, Zhao","roleAtSource":"Article author"},"source":{"title":"Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.38076","publishedAt":"2026-10-27","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce “Prompt Lottery” to describe differing LLM-as-a-Judge judgments under paraphrastic rephrasings intended to preserve evaluation semantics.","excerpt":"We introduce \\textit{Prompt Lottery}, a phenomenon in which paraphrastic rephrasings produce different LLMaJ judgments.","url":"https://arxiv.org/abs/2610.38076","relationship":"Source-stated"},{"text":"The authors report that prompt rephrasing can change evaluation outcomes and sometimes reverse relative model rankings across their study.","excerpt":"these variations can alter evaluation outcomes and, in some settings, reverse relative model rankings.","url":"https://arxiv.org/abs/2610.38076","relationship":"Source-stated"},{"text":"In a user study with 10 human annotators, the authors report that judges’ agreement degraded 6–9 times more than humans’ under prompt rephrasings, falling below a threshold described as generally accepted in usability studies.","excerpt":"A user study with 10 human annotators shows that LLMaJ judges exhibit 6-9$\\times$ larger agreement degradation under prompt rephrasings than humans, falling below the generally accepted threshold in usability studies.","url":"https://arxiv.org/abs/2610.38076","relationship":"Source-stated"},{"text":"The authors say multi-prompt aggregation improves robustness by emphasizing judgments consistent across prompt variants.","excerpt":"Finally, we show that multi-prompt aggregation substantially improves robustness by emphasizing judgments that remain consistent across prompt variants.","url":"https://arxiv.org/abs/2610.38076","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.38076","checkedAt":"2026-11-12T01:30:20.513Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well does multi-prompt aggregation preserve reliable rankings across the evaluation tasks and judge models studied?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI paper reports a “Prompt Lottery” in LLM-as-a-Judge evaluations","summary":"The authors of “Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing” argue that evaluations using an LLM as a judge can change when the prompt is rephrased. They say this can happen even when the rephrasing is intended to preserve the evaluation’s meaning. They call this variation “Prompt Lottery.”\n\nAcross a study covering diverse evaluation tasks, judge families, and model backbones, the authors report that rephrasing can change evaluation outcomes and, in some settings, reverse relative model rankings. In a user study with 10 human annotators, they report that judges’ agreement degraded 6–9 times more under prompt rephrasing than humans’ agreement, falling below a threshold they describe as generally accepted in usability studies.\n\nThe authors argue that evaluating with a single prompt is not robust enough for ranking models. They report that aggregating judgments across multiple prompt variants improves robustness by emphasizing judgments that stay consistent. The abstract does not specify the study’s task-level results or how the aggregation performs beyond the reported improvement.\n\nHow well does multi-prompt aggregation preserve reliable rankings across the evaluation tasks and judge models studied?","question":"How well does multi-prompt aggregation preserve reliable rankings across the evaluation tasks and judge models studied?","claims":[{"text":"The authors introduce “Prompt Lottery” to describe differing LLM-as-a-Judge judgments under paraphrastic rephrasings intended to preserve evaluation semantics.","excerpt":"We introduce \\textit{Prompt Lottery}, a phenomenon in which paraphrastic rephrasings produce different LLMaJ judgments.","support":"source-stated"},{"text":"The authors report that prompt rephrasing can change evaluation outcomes and sometimes reverse relative model rankings across their study.","excerpt":"these variations can alter evaluation outcomes and, in some settings, reverse relative model rankings.","support":"source-stated"},{"text":"In a user study with 10 human annotators, the authors report that judges’ agreement degraded 6–9 times more than humans’ under prompt rephrasings, falling below a threshold described as generally accepted in usability studies.","excerpt":"A user study with 10 human annotators shows that LLMaJ judges exhibit 6-9$\\times$ larger agreement degradation under prompt rephrasings than humans, falling below the generally accepted threshold in usability studies.","support":"source-stated"},{"text":"The authors say multi-prompt aggregation improves robustness by emphasizing judgments consistent across prompt variants.","excerpt":"Finally, we show that multi-prompt aggregation substantially improves robustness by emphasizing judgments that remain consistent across prompt variants.","support":"source-stated"}],"limitations":["The evidence provided is the paper abstract, not its full methods or results; it does not give task-level findings or details needed to assess how the reported aggregation performs across settings.","The abstract does not identify the usability threshold or provide further detail about the human annotation study."],"mediaId":"9e3d9742e5df55c721f6e0dc72086997d2c956a59055fc2df3bd06ddc09dbbc9","correctionReason":null,"id":"30ec9bd5c0549dac85b4e407f11c4492cadaa385e0b953d53f4b7be82f9d76b7","publicationId":"7e53fff2-f4a1-44fd-8509-7230fba70d23","version":1,"sourceUrl":"https://arxiv.org/abs/2610.38076","sourceTitle":"Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-27","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-30T06:45:29.976Z","createdAt":"2026-11-11T08:26:17.534Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"9e3d9742e5df55c721f6e0dc72086997d2c956a59055fc2df3bd06ddc09dbbc9","kind":"article","url":"https://arxiv.org/abs/2610.38076","sourceUrl":"https://arxiv.org/abs/2610.38076","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-31T05:18:34.115Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.38076","checkedAt":"2026-11-12T01:30:20.513Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"9e3d9742e5df55c721f6e0dc72086997d2c956a59055fc2df3bd06ddc09dbbc9","confidence":0.81,"provider":"typesafe","model":"jev-1.13.0","inputHash":"95a71cd269687241ecf7643c83a978e9d946e85834b49ca061c109852725fff1","decidedAt":"2026-10-31T05:18:35.826Z"},"provenance":{"textHash":"60f0a5492e3d5273f762a65904d086e186fbac666b7885749d54b30d450c2d5c","observationId":"af7bd3e3876224dca629bae81bd40e2fa6208218a5c4a30943e3717cf013f6e5","retrievedAt":"2026-10-31T05:18:33.876Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"68efd398-3c4d-431f-868c-9b2a3f00cc5f"},"reviewer":{"model":"gpt-6-luna","runId":"fdcc0b69-7891-45c6-80e1-240fe7541e7a"}},"audit":{"status":"passed","reviewedAt":"2026-11-11T08:26:17.534Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Subasic, Ilija","url":null,"kind":"unknown"},{"name":"Levert, Brett","url":null,"kind":"unknown"},{"name":"Rabinovich, Andrew","url":null,"kind":"unknown"},{"name":"Pakdamansavoji, Sajjad","url":null,"kind":"unknown"},{"name":"Chen, Zhao","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Subasic, Ilija","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Subasic, Ilija","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38076","locator":"head/meta[name=author|citation_author]"},"assertionId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-31T05:18:33.876Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b","sourceUrl":"https://arxiv.org/abs/2610.38076","contentSha256":"f0890cc497428439e7555ac774d25755d3e9cdcbc6fd27d86f08bf2c2b1b15e7","textHash":"60f0a5492e3d5273f762a65904d086e186fbac666b7885749d54b30d450c2d5c","observedAt":"2026-10-31T05:18:33.876Z"}},{"actor":{"name":"Levert, Brett","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Levert, Brett","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38076","locator":"head/meta[name=author|citation_author]"},"assertionId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-31T05:18:33.876Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b","sourceUrl":"https://arxiv.org/abs/2610.38076","contentSha256":"f0890cc497428439e7555ac774d25755d3e9cdcbc6fd27d86f08bf2c2b1b15e7","textHash":"60f0a5492e3d5273f762a65904d086e186fbac666b7885749d54b30d450c2d5c","observedAt":"2026-10-31T05:18:33.876Z"}},{"actor":{"name":"Rabinovich, Andrew","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Rabinovich, Andrew","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38076","locator":"head/meta[name=author|citation_author]"},"assertionId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-31T05:18:33.876Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b","sourceUrl":"https://arxiv.org/abs/2610.38076","contentSha256":"f0890cc497428439e7555ac774d25755d3e9cdcbc6fd27d86f08bf2c2b1b15e7","textHash":"60f0a5492e3d5273f762a65904d086e186fbac666b7885749d54b30d450c2d5c","observedAt":"2026-10-31T05:18:33.876Z"}},{"actor":{"name":"Pakdamansavoji, Sajjad","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Pakdamansavoji, Sajjad","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38076","locator":"head/meta[name=author|citation_author]"},"assertionId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-31T05:18:33.876Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b","sourceUrl":"https://arxiv.org/abs/2610.38076","contentSha256":"f0890cc497428439e7555ac774d25755d3e9cdcbc6fd27d86f08bf2c2b1b15e7","textHash":"60f0a5492e3d5273f762a65904d086e186fbac666b7885749d54b30d450c2d5c","observedAt":"2026-10-31T05:18:33.876Z"}},{"actor":{"name":"Chen, Zhao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chen, Zhao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38076","locator":"head/meta[name=author|citation_author]"},"assertionId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-31T05:18:33.876Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b","sourceUrl":"https://arxiv.org/abs/2610.38076","contentSha256":"f0890cc497428439e7555ac774d25755d3e9cdcbc6fd27d86f08bf2c2b1b15e7","textHash":"60f0a5492e3d5273f762a65904d086e186fbac666b7885749d54b30d450c2d5c","observedAt":"2026-10-31T05:18:33.876Z"}}],"participationCoverage":{"total":5,"returned":5,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.38076","snapshotId":"009b9536935987471e32540f7a37325e5580eaefe87b67aafc74bf72cfd0b91b","contentSha256":"f0890cc497428439e7555ac774d25755d3e9cdcbc6fd27d86f08bf2c2b1b15e7","textHash":"60f0a5492e3d5273f762a65904d086e186fbac666b7885749d54b30d450c2d5c","observedAt":"2026-10-31T05:18:33.876Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/30ec9bd5c0549dac85b4e407f11c4492cadaa385e0b953d53f4b7be82f9d76b7/source"}}}],"audits":[{"publicationId":"7e53fff2-f4a1-44fd-8509-7230fba70d23","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the findings to the authors of the named arXiv paper and keeps them as reported claims. The title's 'arXiv cs.AI' label matches the supplied publisher, though the page lists cs.CL as the primary subject and cs.AI as a cross-list. This is not a material problem.","excerpt":"Authors:Ilija Subasic, Brett Levert, Andrew Rabinovich, Sajjad Pakdamansavoji, Zhao Chen"},{"dimension":"summary","status":"pass","explanation":"The summary leads with the actor and the thesis, explains LLM-as-a-Judge as using an LLM to judge evaluations, and renders 'paraphrastic rephrasings' plainly. It covers ranking reversals, the human comparison, the single-prompt conclusion and the mitigation. Two optional fidelity repairs: it drops the source's 'substantially' before 'improves robustness', and 'judges’ agreement' could say 'LLM judges’ agreement' to avoid confusion with the humans. The question is a genuine open issue.","excerpt":"This indicates that single-prompt evaluation is insufficiently robust for model ranking."},{"dimension":"date","status":"pass","explanation":"The originalPublished value of 2026-10-27 matches the v1 submission date shown on the arXiv page. The later feed date is not used.","excerpt":"[Submitted on 27 Oct 2026]"},{"dimension":"support","status":"pass","explanation":"Each claim is directly supported by the abstract and presented as the authors' report. The limitations correctly note that the abstract-only evidence lacks task-level results and does not identify the usability threshold. No unsupported benefits, costs or real-world reliability claims are added.","excerpt":"we find that these variations can alter evaluation outcomes and, in some settings, reverse relative model rankings"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the original arXiv abstract page, which is the most relevant way to reach the source.","excerpt":null}],"createdAt":"2026-11-11T08:26:17.534Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}