Ahn, Junhyung · Nov 3, 2026
arXiv cs.AI: AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory
Markdown · JSON · Original source
Junhyung Ahn’s paper proposes AGORA, a method for combining ratings from multiple judges when their reliability differs and human-labeled evaluation sets are small. The paper argues that averaging ratings gives weak and strong judges equal weight, while Dawid-Skene must estimate each judge’s error table from limited data. AGORA uses item-response theory: it places items on one latent quality scale and learns each judge’s sensitivity and rating thresholds jointly across items, without human labels. In three , the abstract reports that AGORA matches or exceeds Dawid-Skene on large evaluation sets, with a widening advantage as sets shrink toward a few hundred items. It also reports accuracy with more uneven judge pools, and says a label-free dispersion statistic indicates when trust weighting helps. The learned trust weights track each judge’s validity against human ratings. These are reported benchmark findings, not evidence in the abstract of performance in every evaluation setting. The abstract does not name the benchmarks or give detailed results, so it leaves open how broadly the reported advantage applies.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 3, 2026 (2026-11-03).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T00:07:54.792Z. First observed: 2026-11-05T05:00:20.690Z. Retrieved: 2026-11-09T14:18:26.285Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The abstract does not name the three or provide detailed results, limiting what can be assessed here about the reported performance. The paper’s findings are described as benchmark results; the abstract does not establish performance in other evaluation settings.
Open question
What do the results on each of the three show as evaluation sets shrink toward a few hundred items?
Source-stated
The paper argues that averaging judge ratings can give judges with different reliability equal weight, while Dawid-Skene must estimate each judge’s error table from limited human-labeled data.
the per-item mean weights weak judges as heavily as strong ones, and confusion-table models such as Dawid-Skene must estimate a per-judge error table from too few items
Source-stated
AGORA uses item-response theory to place items on one latent quality scale and jointly learn judge sensitivity and rating thresholds without human labels.
an item-response-theory aggregator that places every item on one latent quality scale and describes each judge by a sensitivity parameter and ordered rating thresholds
Source-stated
Across three benchmarks, the paper reports that AGORA matches or exceeds Dawid-Skene on large evaluation sets, with its advantage widening as sets shrink toward a few hundred items.
Across three benchmarks, AGORA matches or exceeds Dawid-Skene on large evaluation sets, and its advantage widens as the set shrinks toward a few hundred items.
Source-stated
The abstract reports that AGORA remains accurate with more uneven judge pools, and that a label-free dispersion statistic indicates when trust weighting helps.
It remains accurate as pools grow more uneven, a label-free dispersion statistic indicates in advance when trust weighting helps
Source-stated
The paper says its learned trust weights track each judge’s validity against human ratings.
its learned trust weights track per-judge human validity.
Source and contributor context
Ahn, Junhyung · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: a28cf0c1-45ad-4797-838a-56c5404188e6.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T00:07:54.792Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 039a933d2237507cc0021d835c168a8de3ca17ebc1b8118db356cb7de2dc057c. Source text hash: b8c20cd50675e8ba0a7cf6540090cebb215ae944e906e49cd9c9d1a50ff8e834.
Revision and source-check history
Version 1 · 2026-11-10T00:07:54.792Z
arXiv cs.AI: AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory
Junhyung Ahn’s paper proposes AGORA, a method for combining ratings from multiple judges when their reliability differs and human-labeled evaluation sets are small. The paper argues that averaging ratings gives weak and strong judges equal weight, while Dawid-Skene must estimate each judge’s error table from limited data. AGORA uses item-response theory: it places items on one latent quality scale and learns each judge’s sensitivity and rating thresholds jointly across items, without human labels. In three , the abstract reports that AGORA matches or exceeds Dawid-Skene on large evaluation sets, with a widening advantage as sets shrink toward a few hundred items. It also reports accuracy with more uneven judge pools, and says a label-free dispersion statistic indicates when trust weighting helps. The learned trust weights track each judge’s validity against human ratings. These are reported benchmark findings, not evidence in the abstract of performance in every evaluation setting. The abstract does not name the benchmarks or give detailed results, so it leaves open how broadly the reported advantage applies.
Original source
cs.AI · Article
Source details
Original source date: Nov 3, 2026 (2026-11-03).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T00:07:54.792Z. First observed: 2026-11-05T05:00:20.690Z. Retrieved: 2026-11-09T14:18:26.285Z.
Limits
The abstract does not name the three or provide detailed results, limiting what can be assessed here about the reported performance. The paper’s findings are described as benchmark results; the abstract does not establish performance in other evaluation settings.
Open question
What do the results on each of the three show as evaluation sets shrink toward a few hundred items?
Source-stated
The paper argues that averaging judge ratings can give judges with different reliability equal weight, while Dawid-Skene must estimate each judge’s error table from limited human-labeled data.
the per-item mean weights weak judges as heavily as strong ones, and confusion-table models such as Dawid-Skene must estimate a per-judge error table from too few items
Source-stated
AGORA uses item-response theory to place items on one latent quality scale and jointly learn judge sensitivity and rating thresholds without human labels.
an item-response-theory aggregator that places every item on one latent quality scale and describes each judge by a sensitivity parameter and ordered rating thresholds
Source-stated
Across three benchmarks, the paper reports that AGORA matches or exceeds Dawid-Skene on large evaluation sets, with its advantage widening as sets shrink toward a few hundred items.
Across three benchmarks, AGORA matches or exceeds Dawid-Skene on large evaluation sets, and its advantage widens as the set shrinks toward a few hundred items.
Source-stated
The abstract reports that AGORA remains accurate with more uneven judge pools, and that a label-free dispersion statistic indicates when trust weighting helps.
It remains accurate as pools grow more uneven, a label-free dispersion statistic indicates in advance when trust weighting helps
Source-stated
The paper says its learned trust weights track each judge’s validity against human ratings.
its learned trust weights track per-judge human validity.
Source and contributor context
Ahn, Junhyung · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: a28cf0c1-45ad-4797-838a-56c5404188e6.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T00:07:54.792Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 039a933d2237507cc0021d835c168a8de3ca17ebc1b8118db356cb7de2dc057c. Source text hash: b8c20cd50675e8ba0a7cf6540090cebb215ae944e906e49cd9c9d1a50ff8e834.
uncertain · 2026-11-10T00:07:54.792Z · anthropic/claude-opus-5.5
Publication: a28cf0c1-45ad-4797-838a-56c5404188e6
attribution — pass: The paper is correctly attributed to its sole listed author, Junhyung Ahn, on arXiv cs.AI. The method and results are framed mostly as the paper's or abstract's claims. One summary sentence drops that framing; this is noted under summary.
Authors:Junhyung Ahn
summary — uncertain: The reading is faithful, but it omits essential context for a general reader. It never says that LLM-as-a-judge means using AI models to grade outputs in place of costly human evaluation, which is why the work matters. Central terms are left unexplained or need tooltips: LLM judges, item-response theory, latent quality scale, Dawid-Skene and dispersion statistic. Also, "The learned trust weights track each judge's validity against human ratings" reads as established fact. Smallest repairs: add one sentence on the human-evaluation substitute, and prefix that sentence with "The abstract says".
LLM-as-a-judge is a common substitute for costly human evaluation, and aggregating several judges is the standard remedy for any single judge's bias.
date — pass: The originalPublished value of 2026-11-03 matches the v1 submission date. The feed date of 2026-11-05 is a later announcement and is not used.
[Submitted on 3 Nov 2026]
support — pass: Every claim matches the abstract. This covers the critique of mean averaging and Dawid-Skene, the IRT design learned without human labels, and the results across three benchmarks with a widening small-set advantage. It also covers robustness to uneven pools, the label-free dispersion statistic and trust weights tracking human validity. The limitations and the question correctly note that the benchmarks and detailed results are not given, without overstating real-world reliability.
so judge reliability is learned from all items jointly and without human labels
media — pass: The selected media is the verified direct link to the original arXiv abstract page, which is the most direct way to access the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A language model with many learned numerical settings, used to process and generate language.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference