# arXiv cs.AI: Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge [Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-grounded-but-not-immune-confirmation-bias-in-agent-b4157dc3) The authors of “Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge” argue that an Agent-as-a-Judge can gather decisive evidence and still be swayed by a false note attributed to an authority. Across four judges and five datasets, such a note overturned up to 40% of tool-using judges’ correct verdicts, even when the judges collected decisive evidence. The authors treat the note as a controlled diagnostic of confirmation bias, not an adversarial attack. They identify three ways the shift can happen: over-trust, when a judge verifies evidence and then defers; hallucination, when it fabricates support for changing its verdict; and under-investigation, when it stops at the first corroborating signal. For the two Claude judges, the authors say over-trust accounted for most harmful flips; smaller judges showed all three failure modes. Lightweight mitigations reduced the bias but did not remove it. The result raises a practical concern for people relying on these judges to evaluate open-ended agent tasks: collecting evidence alone may not prevent an irrelevant authority cue from changing a verdict. The abstract does not specify which cue properties drive the shift or how the mitigations were tested. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Nov 3, 2026 (2026-11-03). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-09T10:23:16.555Z. First observed: 2026-11-05T05:06:20.581Z. Retrieved: 2026-11-09T10:22:30.064Z. - [Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge](https://arxiv.org/abs/2611.05662) ## Limits The source is marked “Preprint. Under Review”; the evidence provided is the abstract, not the full paper. The abstract says the shift is driven by specific properties of the cue but does not identify those properties or describe how the lightweight mitigations were evaluated. The amount by which they reduce bias is not stated. ## Open question Which properties of the false note drive the change in verdict, and how much do the lightweight mitigations reduce it? ## Source-stated The authors report that a false pre-screening note attributed to an authority overturned up to 40% of tool-using judges’ correct verdicts across four judges and five datasets, despite decisive evidence collection. > a single false pre-screening note, attributed to an authority and carrying no evidential bearing on the correct verdict, overturns up to 40% of a tool-using judge's correct verdicts across four judges and five datasets, even though the judge collects decisive evidence. - [Supporting source](https://arxiv.org/abs/2611.05662) ## Source-stated The authors identify over-trust, hallucination, and under-investigation as three failure modes through which confirmation bias changes judge behavior. > We classify changes in judge behavior and attribute changes to one of three failure modes: over-trust (verify, conclude correctly, then defer), hallucination (fabricate support for the flip), and under-investigation (stop at the first corroborating signal). - [Supporting source](https://arxiv.org/abs/2611.05662) ## Source-stated The authors say over-trust predominated for the two Claude judges, while smaller judges showed all three error modes; lightweight mitigations blunted but did not remove the bias. > For the two Claude judges, the failure is predominantly over-trust (67-75% of harmful flips), while smaller judges have error modes spread across all three. Lightweight mitigations blunt but never remove the bias. - [Supporting source](https://arxiv.org/abs/2611.05662) ## Source and contributor context Brown, Drake Benjamin, Pahwa, Khushbu, Ge, Yubin, Romeo, Salvatore, Agnihotri, Akhil, Zhang, Yi, Ma, Xiaofei, Bonadiman, Daniele · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Hallucination When an AI system states something false or made up as if it were true. - [Definition reference](https://developers.google.com/machine-learning/glossary#hallucination) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Claude Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding. - [Explore this context](/entities/claude) - [Reference](https://claude.com/product/overview) - [Reference](https://www.anthropic.com/news) - [Reference](https://www.anthropic.com/research/team/economics) ## Editorial provenance Version 1. Publication ID: dc1f9f97-2449-4f70-8c4f-c57477026861. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T10:23:16.555Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 5aef6733a2c742849e31c62e79a6e98fceae5cf8c511bf09e5ec577f2940c2fd. Source text hash: 4981375b641273c9b3a7ce8bde47e22f658403c43eea5f2f636ab38d17c2a008. ## Revision and source-check history ### Version 1 — 2026-11-09T10:23:16.555Z The authors of “Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge” argue that an Agent-as-a-Judge can gather decisive evidence and still be swayed by a false note attributed to an authority. Across four judges and five datasets, such a note overturned up to 40% of tool-using judges’ correct verdicts, even when the judges collected decisive evidence. The authors treat the note as a controlled diagnostic of confirmation bias, not an adversarial attack. They identify three ways the shift can happen: over-trust, when a judge verifies evidence and then defers; hallucination, when it fabricates support for changing its verdict; and under-investigation, when it stops at the first corroborating signal. For the two Claude judges, the authors say over-trust accounted for most harmful flips; smaller judges showed all three failure modes. Lightweight mitigations reduced the bias but did not remove it. The result raises a practical concern for people relying on these judges to evaluate open-ended agent tasks: collecting evidence alone may not prevent an irrelevant authority cue from changing a verdict. The abstract does not specify which cue properties drive the shift or how the mitigations were tested. ## Original source arXiv cs.AI · Article Original source date: Nov 3, 2026 (2026-11-03). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-09T10:23:16.555Z. First observed: 2026-11-05T05:06:20.581Z. Retrieved: 2026-11-09T10:22:30.064Z. - [Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge](https://arxiv.org/abs/2611.05662) ## Limits The source is marked “Preprint. Under Review”; the evidence provided is the abstract, not the full paper. The abstract says the shift is driven by specific properties of the cue but does not identify those properties or describe how the lightweight mitigations were evaluated. The amount by which they reduce bias is not stated. ## Open question Which properties of the false note drive the change in verdict, and how much do the lightweight mitigations reduce it? ## Source-stated The authors report that a false pre-screening note attributed to an authority overturned up to 40% of tool-using judges’ correct verdicts across four judges and five datasets, despite decisive evidence collection. > a single false pre-screening note, attributed to an authority and carrying no evidential bearing on the correct verdict, overturns up to 40% of a tool-using judge's correct verdicts across four judges and five datasets, even though the judge collects decisive evidence. - [Supporting source](https://arxiv.org/abs/2611.05662) ## Source-stated The authors identify over-trust, hallucination, and under-investigation as three failure modes through which confirmation bias changes judge behavior. > We classify changes in judge behavior and attribute changes to one of three failure modes: over-trust (verify, conclude correctly, then defer), hallucination (fabricate support for the flip), and under-investigation (stop at the first corroborating signal). - [Supporting source](https://arxiv.org/abs/2611.05662) ## Source-stated The authors say over-trust predominated for the two Claude judges, while smaller judges showed all three error modes; lightweight mitigations blunted but did not remove the bias. > For the two Claude judges, the failure is predominantly over-trust (67-75% of harmful flips), while smaller judges have error modes spread across all three. Lightweight mitigations blunt but never remove the bias. - [Supporting source](https://arxiv.org/abs/2611.05662) ## Source and contributor context Brown, Drake Benjamin, Pahwa, Khushbu, Ge, Yubin, Romeo, Salvatore, Agnihotri, Akhil, Zhang, Yi, Ma, Xiaofei, Bonadiman, Daniele · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Hallucination When an AI system states something false or made up as if it were true. - [Definition reference](https://developers.google.com/machine-learning/glossary#hallucination) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Claude Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding. - [Explore this context](/entities/claude) - [Reference](https://claude.com/product/overview) - [Reference](https://www.anthropic.com/news) - [Reference](https://www.anthropic.com/research/team/economics) ## Editorial provenance Version 1. Publication ID: dc1f9f97-2449-4f70-8c4f-c57477026861. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T10:23:16.555Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 5aef6733a2c742849e31c62e79a6e98fceae5cf8c511bf09e5ec577f2940c2fd. Source text hash: 4981375b641273c9b3a7ce8bde47e22f658403c43eea5f2f636ab38d17c2a008. ### uncertain — 2026-11-09T10:23:16.555Z — anthropic/claude-opus-5.5 Publication: dc1f9f97-2449-4f70-8c4f-c57477026861 attribution — pass: The findings, failure-mode taxonomy, Claude-judge breakdown, and mitigation result are attributed to the paper's authors. The source is correctly identified as an arXiv cs.AI preprint, and the 'Preprint. Under Review' status is noted in the limitations. > Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge summary — uncertain: The reading is faithful but never says what an Agent-as-a-Judge is: an AI evaluator given tools and another agent's full trajectory so it can check a result against the environment. A general reader may not realize an AI is grading another AI's work, and 'trajectory' and 'tool-using' have no plain gloss. Smallest repair: add one plain sentence defining the term using the abstract's own description, and suggest a term-library tooltip. The title only prefixes the source name, but that is acceptable. The question is substantive. > an Agent-as-a-Judge: an evaluator handed tools and the agent's full trajectory so it can verify a result against the environment date — pass: originalPublished is 2026-11-03, which matches the arXiv v1 submission date. It is not the later feed date of Nov 5. > \[Submitted on 3 Nov 2026\] support — pass: The claims match the abstract: up to 40% of correct verdicts overturned across four judges and five datasets; the note framed as a diagnostic, not an attack; three failure modes; over-trust predominating (67-75%) for the Claude judges; and mitigations that blunt but never remove the bias. The practical concern is framed as a possibility. The stated gaps about cue properties and mitigation testing are accurate. > Lightweight mitigations blunt but never remove the bias. media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/b4157dc37d84a4ed3fe269c947043b87229c4f3487e2dbd1953369e28eefad63/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-grounded-but-not-immune-confirmation-bias-in-agent-b4157dc3.json)