Brown, Drake Benjamin, Pahwa, Khushbu, Ge, Yubin, Romeo, Salvatore, Agnihotri, Akhil, Zhang, Yi, Ma, Xiaofei, Bonadiman, Daniele · Nov 3, 2026
arXiv cs.AI: Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge
Markdown · JSON · Original source
The authors of “Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge” argue that an Agent-as-a-Judge can gather decisive evidence and still be swayed by a false note attributed to an authority. Across four judges and five datasets, such a note overturned up to 40% of tool-using judges’ correct verdicts, even when the judges collected decisive evidence. The authors treat the note as a controlled diagnostic of confirmation bias, not an adversarial attack. They identify three ways the shift can happen: over-trust, when a judge verifies evidence and then defers; , when it fabricates support for changing its verdict; and under-investigation, when it stops at the first corroborating signal. For the two judges, the authors say over-trust accounted for most harmful flips; smaller judges showed all three failure modes. Lightweight mitigations reduced the bias but did not remove it. The result raises a practical concern for people relying on these judges to evaluate open-ended agent tasks: collecting evidence alone may not prevent an irrelevant authority cue from changing a verdict. The abstract does not specify which cue properties drive the shift or how the mitigations were tested.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 3, 2026 (2026-11-03).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T10:23:16.555Z. First observed: 2026-11-05T05:06:20.581Z. Retrieved: 2026-11-09T10:22:30.064Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The source is marked “Preprint. Under Review”; the evidence provided is the abstract, not the full paper. The abstract says the shift is driven by specific properties of the cue but does not identify those properties or describe how the lightweight mitigations were evaluated. The amount by which they reduce bias is not stated.
Open question
Which properties of the false note drive the change in verdict, and how much do the lightweight mitigations reduce it?
Source-stated
The authors report that a false pre-screening note attributed to an authority overturned up to 40% of tool-using judges’ correct verdicts across four judges and five datasets, despite decisive evidence collection.
a single false pre-screening note, attributed to an authority and carrying no evidential bearing on the correct verdict, overturns up to 40% of a tool-using judge's correct verdicts across four judges and five datasets, even though the judge collects decisive evidence.
Source-stated
The authors identify over-trust, hallucination, and under-investigation as three failure modes through which confirmation bias changes judge behavior.
We classify changes in judge behavior and attribute changes to one of three failure modes: over-trust (verify, conclude correctly, then defer), hallucination (fabricate support for the flip), and under-investigation (stop at the first corroborating signal).
Source-stated
The authors say over-trust predominated for the two Claude judges, while smaller judges showed all three error modes; lightweight mitigations blunted but did not remove the bias.
For the two Claude judges, the failure is predominantly over-trust (67-75% of harmful flips), while smaller judges have error modes spread across all three. Lightweight mitigations blunt but never remove the bias.
Source and contributor context
Brown, Drake Benjamin, Pahwa, Khushbu, Ge, Yubin, Romeo, Salvatore, Agnihotri, Akhil, Zhang, Yi, Ma, Xiaofei, Bonadiman, Daniele · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Hallucination
When an AI system states something false or made up as if it were true.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: dc1f9f97-2449-4f70-8c4f-c57477026861.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T10:23:16.555Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 5aef6733a2c742849e31c62e79a6e98fceae5cf8c511bf09e5ec577f2940c2fd. Source text hash: 4981375b641273c9b3a7ce8bde47e22f658403c43eea5f2f636ab38d17c2a008.
Revision and source-check history
Version 1 · 2026-11-09T10:23:16.555Z
arXiv cs.AI: Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge
The authors of “Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge” argue that an Agent-as-a-Judge can gather decisive evidence and still be swayed by a false note attributed to an authority. Across four judges and five datasets, such a note overturned up to 40% of tool-using judges’ correct verdicts, even when the judges collected decisive evidence. The authors treat the note as a controlled diagnostic of confirmation bias, not an adversarial attack. They identify three ways the shift can happen: over-trust, when a judge verifies evidence and then defers; , when it fabricates support for changing its verdict; and under-investigation, when it stops at the first corroborating signal. For the two judges, the authors say over-trust accounted for most harmful flips; smaller judges showed all three failure modes. Lightweight mitigations reduced the bias but did not remove it. The result raises a practical concern for people relying on these judges to evaluate open-ended agent tasks: collecting evidence alone may not prevent an irrelevant authority cue from changing a verdict. The abstract does not specify which cue properties drive the shift or how the mitigations were tested.
Original source
cs.AI · Article
Source details
Original source date: Nov 3, 2026 (2026-11-03).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T10:23:16.555Z. First observed: 2026-11-05T05:06:20.581Z. Retrieved: 2026-11-09T10:22:30.064Z.
Limits
The source is marked “Preprint. Under Review”; the evidence provided is the abstract, not the full paper. The abstract says the shift is driven by specific properties of the cue but does not identify those properties or describe how the lightweight mitigations were evaluated. The amount by which they reduce bias is not stated.
Open question
Which properties of the false note drive the change in verdict, and how much do the lightweight mitigations reduce it?
Source-stated
The authors report that a false pre-screening note attributed to an authority overturned up to 40% of tool-using judges’ correct verdicts across four judges and five datasets, despite decisive evidence collection.
a single false pre-screening note, attributed to an authority and carrying no evidential bearing on the correct verdict, overturns up to 40% of a tool-using judge's correct verdicts across four judges and five datasets, even though the judge collects decisive evidence.
Source-stated
The authors identify over-trust, hallucination, and under-investigation as three failure modes through which confirmation bias changes judge behavior.
We classify changes in judge behavior and attribute changes to one of three failure modes: over-trust (verify, conclude correctly, then defer), hallucination (fabricate support for the flip), and under-investigation (stop at the first corroborating signal).
Source-stated
The authors say over-trust predominated for the two Claude judges, while smaller judges showed all three error modes; lightweight mitigations blunted but did not remove the bias.
For the two Claude judges, the failure is predominantly over-trust (67-75% of harmful flips), while smaller judges have error modes spread across all three. Lightweight mitigations blunt but never remove the bias.
Source and contributor context
Brown, Drake Benjamin, Pahwa, Khushbu, Ge, Yubin, Romeo, Salvatore, Agnihotri, Akhil, Zhang, Yi, Ma, Xiaofei, Bonadiman, Daniele · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Hallucination
When an AI system states something false or made up as if it were true.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: dc1f9f97-2449-4f70-8c4f-c57477026861.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T10:23:16.555Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 5aef6733a2c742849e31c62e79a6e98fceae5cf8c511bf09e5ec577f2940c2fd. Source text hash: 4981375b641273c9b3a7ce8bde47e22f658403c43eea5f2f636ab38d17c2a008.
uncertain · 2026-11-09T10:23:16.555Z · anthropic/claude-opus-5.5
Publication: dc1f9f97-2449-4f70-8c4f-c57477026861
attribution — pass: The findings, failure-mode taxonomy, Claude-judge breakdown, and mitigation result are attributed to the paper's authors. The source is correctly identified as an arXiv cs.AI preprint, and the 'Preprint. Under Review' status is noted in the limitations.
Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge
summary — uncertain: The reading is faithful but never says what an Agent-as-a-Judge is: an AI evaluator given tools and another agent's full trajectory so it can check a result against the environment. A general reader may not realize an AI is grading another AI's work, and 'trajectory' and 'tool-using' have no plain gloss. Smallest repair: add one plain sentence defining the term using the abstract's own description, and suggest a term-library tooltip. The title only prefixes the source name, but that is acceptable. The question is substantive.
an Agent-as-a-Judge: an evaluator handed tools and the agent's full trajectory so it can verify a result against the environment
date — pass: originalPublished is 2026-11-03, which matches the arXiv v1 submission date. It is not the later feed date of Nov 5.
[Submitted on 3 Nov 2026]
support — pass: The claims match the abstract: up to 40% of correct verdicts overturned across four judges and five datasets; the note framed as a diagnostic, not an attack; three failure modes; over-trust predominating (67-75%) for the Claude judges; and mitigations that blunt but never remove the bias. The practical concern is framed as a possibility. The stated gaps about cue properties and mitigation testing are accurate.
Lightweight mitigations blunt but never remove the bias.
media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
When an AI system states something false or made up as if it were true.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivWhen an AI system states something false or made up as if it were true.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv