Paschali, Magdalini, Aali, Asad, Koirala, Arogya, Johnston, Andrew, McNamara, Katherine, Gaus, Richard, Daneshjou, Roxana, Larson, David B., Chaudhari, Akshay S. · Nov 6, 2026
arXiv cs.AI: Adversarial Evaluation of Large Language Models for Patient-Friendly Radiology Report Summarization
Markdown · JSON · Original source
The authors of this study evaluate whether (LLMs) preserve clinical details when turning radiology reports into patient-friendly summaries, including when report wording is deliberately changed or information is removed. They tested six LLMs on CT and MRI reports, using two prompt styles and 18,000 evaluation pairs. Their results distinguish between kinds of edits: impression removals were handled more robustly than term substitutions. GPT-5-mini had the strongest scores for factual faithfulness and certainty fidelity, yet its summaries still missed changed anatomical terms, diagnostic certainty, and severity descriptors in some cases. The authors argue that adversarial evaluation can expose clinically meaningful failures. For patients, those omissions could matter because a summary may not reflect a detail or degree of certainty present in the report; the study evaluates summaries rather than patient outcomes. The authors also report that agreement between their automated judge and radiologists was comparable to agreement between two radiologists. The work was submitted to Radiology: Artificial Intelligence for possible publication; the evidence here is its abstract.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 6, 2026 (2026-11-06).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-12T11:03:52.633Z. First observed: 2026-11-10T08:42:26.467Z. Retrieved: 2026-11-12T11:02:32.985Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The evidence is the paper's abstract; it states that the work was submitted to Radiology: Artificial Intelligence for possible publication. The evaluation uses adversarial variants of CT and MRI reports; the abstract does not establish how often those edits occur in routine reports. The abstract reports evaluation scores and agreement, not effects on patients or clinical outcomes.
Open question
How often do the adversarial edits used in this evaluation reflect changes or omissions found in real radiology reports?
Source-stated
The authors evaluated six LLMs on adversarially modified CT and MRI reports, producing 18,000 evaluation pairs.
Six LLMs summarized original and adversarial impressions using 2 prompt styles, yielding 18,000 evaluation pairs.
Source-stated
Impression-removal edits were handled more robustly than term substitutions.
Impression-removal edits were handled more robustly than term substitutions.
Source-stated
GPT-5-mini achieved the highest factual faithfulness and certainty fidelity under both types of adversarial edits, but still missed some substitutions.
GPT-5-mini achieved the highest factual faithfulness and certainty fidelity under both adversarial edits.
Source-stated
For GPT-5-mini summaries, anatomical terms were not reflected in 23.7% of cases, diagnostic certainty in 22.3%, and descriptor terms in 21.2%.
substitutions of anatomical terms were not reflected in 23.7% of summaries, diagnostic certainty in 22.3%, and descriptor terms (e.g., "substantial") in 21.2%.
Source-stated
The authors conclude that adversarial evaluation exposes clinically meaningful failure modes in patient-friendly radiology summarization.
Adversarial evaluation exposes clinically meaningful failure modes, including missed anatomical modifiers, altered diagnostic certainty, and omitted severity descriptors.
Source-stated
The authors report that the automated judge's agreement with two radiologists was comparable to agreement between radiologists.
Judge-radiologist agreement (macro-F1: 0.55-0.56) was comparable to inter-radiologist agreement (0.53).
Source and contributor context
Paschali, Magdalini, Aali, Asad, Koirala, Arogya, Johnston, Andrew, McNamara, Katherine, Gaus, Richard, Daneshjou, Roxana, Larson, David B., Chaudhari, Akshay S. · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: ed5f7324-99e6-4ae9-8069-4149c48aaf73.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-12T11:03:52.633Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: e8c5f5ec2f2213021b902fd6e06aa28d0ed8eb12d5e1bb97d7eddd47d968dd1d. Source text hash: 70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9.
Revision and source-check history
Version 1 · 2026-11-12T11:03:52.633Z
arXiv cs.AI: Adversarial Evaluation of Large Language Models for Patient-Friendly Radiology Report Summarization
The authors of this study evaluate whether (LLMs) preserve clinical details when turning radiology reports into patient-friendly summaries, including when report wording is deliberately changed or information is removed. They tested six LLMs on CT and MRI reports, using two prompt styles and 18,000 evaluation pairs. Their results distinguish between kinds of edits: impression removals were handled more robustly than term substitutions. GPT-5-mini had the strongest scores for factual faithfulness and certainty fidelity, yet its summaries still missed changed anatomical terms, diagnostic certainty, and severity descriptors in some cases. The authors argue that adversarial evaluation can expose clinically meaningful failures. For patients, those omissions could matter because a summary may not reflect a detail or degree of certainty present in the report; the study evaluates summaries rather than patient outcomes. The authors also report that agreement between their automated judge and radiologists was comparable to agreement between two radiologists. The work was submitted to Radiology: Artificial Intelligence for possible publication; the evidence here is its abstract.
Original source
cs.AI · Article
Source details
Original source date: Nov 6, 2026 (2026-11-06).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-12T11:03:52.633Z. First observed: 2026-11-10T08:42:26.467Z. Retrieved: 2026-11-12T11:02:32.985Z.
Limits
The evidence is the paper's abstract; it states that the work was submitted to Radiology: Artificial Intelligence for possible publication. The evaluation uses adversarial variants of CT and MRI reports; the abstract does not establish how often those edits occur in routine reports. The abstract reports evaluation scores and agreement, not effects on patients or clinical outcomes.
Open question
How often do the adversarial edits used in this evaluation reflect changes or omissions found in real radiology reports?
Source-stated
The authors evaluated six LLMs on adversarially modified CT and MRI reports, producing 18,000 evaluation pairs.
Six LLMs summarized original and adversarial impressions using 2 prompt styles, yielding 18,000 evaluation pairs.
Source-stated
Impression-removal edits were handled more robustly than term substitutions.
Impression-removal edits were handled more robustly than term substitutions.
Source-stated
GPT-5-mini achieved the highest factual faithfulness and certainty fidelity under both types of adversarial edits, but still missed some substitutions.
GPT-5-mini achieved the highest factual faithfulness and certainty fidelity under both adversarial edits.
Source-stated
For GPT-5-mini summaries, anatomical terms were not reflected in 23.7% of cases, diagnostic certainty in 22.3%, and descriptor terms in 21.2%.
substitutions of anatomical terms were not reflected in 23.7% of summaries, diagnostic certainty in 22.3%, and descriptor terms (e.g., "substantial") in 21.2%.
Source-stated
The authors conclude that adversarial evaluation exposes clinically meaningful failure modes in patient-friendly radiology summarization.
Adversarial evaluation exposes clinically meaningful failure modes, including missed anatomical modifiers, altered diagnostic certainty, and omitted severity descriptors.
Source-stated
The authors report that the automated judge's agreement with two radiologists was comparable to agreement between radiologists.
Judge-radiologist agreement (macro-F1: 0.55-0.56) was comparable to inter-radiologist agreement (0.53).
Source and contributor context
Paschali, Magdalini, Aali, Asad, Koirala, Arogya, Johnston, Andrew, McNamara, Katherine, Gaus, Richard, Daneshjou, Roxana, Larson, David B., Chaudhari, Akshay S. · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: ed5f7324-99e6-4ae9-8069-4149c48aaf73.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-12T11:03:52.633Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: e8c5f5ec2f2213021b902fd6e06aa28d0ed8eb12d5e1bb97d7eddd47d968dd1d. Source text hash: 70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9.
passed · 2026-11-12T11:03:52.633Z · anthropic/claude-opus-5.5
Publication: ed5f7324-99e6-4ae9-8069-4149c48aaf73
attribution — pass: The reading attributes the framework, results and conclusion to the paper's authors, and correctly names arXiv cs.AI as the publisher. Model results are reported as findings, not presented as EM's own claims.
Authors:Magdalini Paschali, Asad Aali, Arogya Koirala, Andrew Johnston, Katherine McNamara, Richard Gaus, Roxana Daneshjou, David B. Larson, Akshay S. Chaudhari
summary — pass: The reading accurately covers the setup (six LLMs, two prompt styles, 18,000 pairs), the comparison of edit types, GPT-5-mini's lead and its remaining misses, judge agreement and submission status. 'Adversarial' is explained as deliberately changing wording or removing information. 'Impression' (the conclusion section of a radiology report) is not explained but does not block understanding; it is a candidate for a term-library tooltip. The omitted hallucination-rate comparison does not distort the reading. The question is a genuine open issue.
Impression-removal edits were handled more robustly than term substitutions.
date — pass: originalPublished 2026-11-06 matches the v1 submission date on arXiv. The feed date of Nov 10 is correctly treated as later.
[Submitted on 6 Nov 2026]
support — pass: All claims match the abstract. The abstract attributes the 23.7/22.3/21.2% figures to 'the highest-scoring summarizer'. The reading assigns them to GPT-5-mini, which fits the abstract's statement that GPT-5-mini scored highest. The patient-impact sentence is hedged ('could matter') and explicitly separates summary evaluation from patient outcomes. It is grounded in the authors' 'clinically meaningful' conclusion. The limitations are accurate.
Even for the highest-scoring summarizer, substitutions of anatomical terms were not reflected in 23.7% of summaries, diagnostic certainty in 22.3%, and descriptor terms (e.g., "substantial") in 21.2%.
media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A language model with many learned numerical settings, used to process and generate language.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv