Tang, Yupeng, Lin, Mingfeng · Nov 12, 2026
arXiv cs.AI: Expert disagreement can reduce LLM decision accuracy
Markdown · JSON · Original source
Yupeng Tang and Mingfeng Lin argue that counting how often language models change their answers cannot show whether they corrected mistakes or abandoned correct decisions. Their study compares a model’s initial answer with its revision after neutral reflection, anonymous disagreement, or disagreement labeled as coming from a domain expert. For binary decisions, they say revision improves accuracy only when wrong answers are reversed often enough relative to correct ones; at 90% initial accuracy, the required ratio is nine to one. Across five models and 1,989 evidence-grounded decisions in medicine, science, and law, none met that threshold under expert disagreement, and accuracy fell by as much as 16.1 points. Yet the expert label increased reversals for four models, while its effect on accuracy ranged from a 5.2-point loss to a 2.4-point gain. The authors also report that identical arguments can prompt more reversals when labeled as expert advice, and that models respond to stated source reliability. Their central warning is that reversal counts alone can mistake deference for sound revision. They recommend evaluating revision pipelines with paired, conditional metrics for each model and protocol.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 12, 2026 (2026-11-12).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-13T11:47:39.774Z. First observed: 2026-11-13T07:43:35.205Z. Retrieved: 2026-11-13T11:46:48.120Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The supplied evidence is the paper’s abstract and does not identify the five models or provide the detailed protocols behind the reported results. The authors describe results across selected evidence-grounded decisions and domains; the abstract does not establish how the findings generalize beyond those evaluations.
Open question
Which model and protocol conditions determine whether an expert label improves accuracy rather than merely increasing reversals?
Source-stated
The authors introduce a controlled two-pass audit that varies the revision context while holding the question, evidence, and initial answer fixed.
holds the question, the evidence, and the model's initial answer fixed and varies only the revision context
Source-stated
For binary decisions, revision raises accuracy only if wrong answers are reversed more often than correct ones by a threshold based on initial accuracy; at 90% initial accuracy, the ratio is nine to one.
reversing wrong answers more than $a/(1-a)$ times as often as correct ones, where $a$ is initial accuracy (nine times at 90%).
Source-stated
Across five models and 1,989 evidence-grounded decisions in medicine, science, and law, no model met the authors’ threshold under expert disagreement, and accuracy fell by up to 16.1 points.
Across five models and 1,989 evidence-grounded decisions in medicine, science, and law, no model clears this bar under expert disagreement, and accuracy falls by up to 16.1 points.
Source-stated
The expert label increased reversals for four models, but its effect on accuracy ranged from a 5.2-point loss to a 2.4-point gain.
The expert label, a prestige cue, adds reversals for four of the five models, yet its effect on accuracy ranges from a 5.2-point loss to a 2.4-point gain
Source-stated
The authors report that an expert label can increase reversals even when the arguments are identical, and recommend paired, conditional evaluation metrics for each model and protocol.
With identical arguments from both sources, the label still raises reversal for three proprietary models and in a pooled analysis of four newer models. Models also respond to stated source reliability. Revision pipelines should therefore be evaluated with paired, conditional metrics, per model and
Source and contributor context
Tang, Yupeng, Lin, Mingfeng · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: f2985812-e799-477f-8e7b-62324fb657ae.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-13T11:47:39.774Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: d747cd853b9c176531224f1de9833296ff273c36ee8a4b48bae927af8645aabe. Source text hash: 2c34b81ec8724f9b29eef3b7b890af7da679ec6ce90474af6e32b37a2ea00dbb.
Revision and source-check history
Version 1 · 2026-11-13T11:47:39.774Z
arXiv cs.AI: Expert disagreement can reduce LLM decision accuracy
Yupeng Tang and Mingfeng Lin argue that counting how often language models change their answers cannot show whether they corrected mistakes or abandoned correct decisions. Their study compares a model’s initial answer with its revision after neutral reflection, anonymous disagreement, or disagreement labeled as coming from a domain expert. For binary decisions, they say revision improves accuracy only when wrong answers are reversed often enough relative to correct ones; at 90% initial accuracy, the required ratio is nine to one. Across five models and 1,989 evidence-grounded decisions in medicine, science, and law, none met that threshold under expert disagreement, and accuracy fell by as much as 16.1 points. Yet the expert label increased reversals for four models, while its effect on accuracy ranged from a 5.2-point loss to a 2.4-point gain. The authors also report that identical arguments can prompt more reversals when labeled as expert advice, and that models respond to stated source reliability. Their central warning is that reversal counts alone can mistake deference for sound revision. They recommend evaluating revision pipelines with paired, conditional metrics for each model and protocol.
Original source
cs.AI · Article
Source details
Original source date: Nov 12, 2026 (2026-11-12).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-13T11:47:39.774Z. First observed: 2026-11-13T07:43:35.205Z. Retrieved: 2026-11-13T11:46:48.120Z.
Limits
The supplied evidence is the paper’s abstract and does not identify the five models or provide the detailed protocols behind the reported results. The authors describe results across selected evidence-grounded decisions and domains; the abstract does not establish how the findings generalize beyond those evaluations.
Open question
Which model and protocol conditions determine whether an expert label improves accuracy rather than merely increasing reversals?
Source-stated
The authors introduce a controlled two-pass audit that varies the revision context while holding the question, evidence, and initial answer fixed.
holds the question, the evidence, and the model's initial answer fixed and varies only the revision context
Source-stated
For binary decisions, revision raises accuracy only if wrong answers are reversed more often than correct ones by a threshold based on initial accuracy; at 90% initial accuracy, the ratio is nine to one.
reversing wrong answers more than $a/(1-a)$ times as often as correct ones, where $a$ is initial accuracy (nine times at 90%).
Source-stated
Across five models and 1,989 evidence-grounded decisions in medicine, science, and law, no model met the authors’ threshold under expert disagreement, and accuracy fell by up to 16.1 points.
Across five models and 1,989 evidence-grounded decisions in medicine, science, and law, no model clears this bar under expert disagreement, and accuracy falls by up to 16.1 points.
Source-stated
The expert label increased reversals for four models, but its effect on accuracy ranged from a 5.2-point loss to a 2.4-point gain.
The expert label, a prestige cue, adds reversals for four of the five models, yet its effect on accuracy ranges from a 5.2-point loss to a 2.4-point gain
Source-stated
The authors report that an expert label can increase reversals even when the arguments are identical, and recommend paired, conditional evaluation metrics for each model and protocol.
With identical arguments from both sources, the label still raises reversal for three proprietary models and in a pooled analysis of four newer models. Models also respond to stated source reliability. Revision pipelines should therefore be evaluated with paired, conditional metrics, per model and
Source and contributor context
Tang, Yupeng, Lin, Mingfeng · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: f2985812-e799-477f-8e7b-62324fb657ae.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-13T11:47:39.774Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: d747cd853b9c176531224f1de9833296ff273c36ee8a4b48bae927af8645aabe. Source text hash: 2c34b81ec8724f9b29eef3b7b890af7da679ec6ce90474af6e32b37a2ea00dbb.
uncertain · 2026-11-13T11:47:39.774Z · anthropic/claude-opus-5.5
Publication: f2985812-e799-477f-8e7b-62324fb657ae
attribution — pass: The reading attributes the study and its claims to authors Yupeng Tang and Mingfeng Lin and frames the results as their reports. The arXiv cs.AI label correctly identifies the publisher context.
Authors:Yupeng Tang, Mingfeng Lin
summary — uncertain: Mostly faithful and readable, but two passages lose scope. First, the summary says identical arguments 'can prompt more reversals when labeled as expert advice.' The source limits this to three proprietary models and a pooled analysis of four newer models. Smallest repair: restore that scope. Second, 'can mistake deference for sound revision' recasts the source's point that reversal counts are a poor proxy for harm, since more reversals sometimes accompanied accuracy gains. Smallest repair: use the source's framing. 'Paired, conditional metrics' is left unexplained, but this is not central.
With identical arguments from both sources, the label still raises reversal for three proprietary models and in a pooled analysis of four newer models.
date — pass: The originalPublished date, 2026-11-12, matches the v1 submission date on the arXiv page. The later feed date of Nov 13 was correctly not used.
[Submitted on 12 Nov 2026]
support — pass: Each claim and its excerpt matches the abstract: the two-pass audit design, the a/(1-a) threshold with nine-to-one at 90%, no model clearing the bar with up to a 16.1-point fall, and the label adding reversals for four models with effects from -5.2 to +2.4 points. Claim 5 is hedged with 'can'. The limitations accurately note that the abstract does not identify the five models.
so reversal counts are a poor proxy for harm
media — pass: The selected media is the verified, direct arXiv abstract page for this paper, which matches the source URL.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv