{"id":"d3185204082cbcdf3edf4e5d4ced394637504f42b703fc6e542f64d48bf548be","title":"arXiv cs.AI: Adversarial Evaluation of Large Language Models for Patient-Friendly Radiology Report Summarization","argument":"The authors of this study evaluate whether large language models (LLMs) preserve clinical details when turning radiology reports into patient-friendly summaries, including when report wording is deliberately changed or information is removed. They tested six LLMs on CT and MRI reports, using two prompt styles and 18,000 evaluation pairs. Their results distinguish between kinds of edits: impression removals were handled more robustly than term substitutions. GPT-5-mini had the strongest scores for factual faithfulness and certainty fidelity, yet its summaries still missed changed anatomical terms, diagnostic certainty, and severity descriptors in some cases. The authors argue that adversarial evaluation can expose clinically meaningful failures. For patients, those omissions could matter because a summary may not reflect a detail or degree of certainty present in the report; the study evaluates summaries rather than patient outcomes. The authors also report that agreement between their automated judge and radiologists was comparable to agreement between two radiologists. The work was submitted to Radiology: Artificial Intelligence for possible publication; the evidence here is its arXiv abstract.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence is the paper's arXiv abstract; it states that the work was submitted to Radiology: Artificial Intelligence for possible publication. The evaluation uses adversarial variants of CT and MRI reports; the abstract does not establish how often those edits occur in routine reports. The abstract reports evaluation scores and agreement, not effects on patients or clinical outcomes.","speaker":{"name":"Paschali, Magdalini, Aali, Asad, Koirala, Arogya, Johnston, Andrew, McNamara, Katherine, Gaus, Richard, Daneshjou, Roxana, Larson, David B., Chaudhari, Akshay S.","roleAtSource":"Article author"},"source":{"title":"Adversarial Evaluation of Large Language Models for Patient-Friendly Radiology Report Summarization","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.10277","publishedAt":"2026-11-06","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors evaluated six LLMs on adversarially modified CT and MRI reports, producing 18,000 evaluation pairs.","excerpt":"Six LLMs summarized original and adversarial impressions using 2 prompt styles, yielding 18,000 evaluation pairs.","url":"https://arxiv.org/abs/2611.10277","relationship":"Source-stated"},{"text":"Impression-removal edits were handled more robustly than term substitutions.","excerpt":"Impression-removal edits were handled more robustly than term substitutions.","url":"https://arxiv.org/abs/2611.10277","relationship":"Source-stated"},{"text":"GPT-5-mini achieved the highest factual faithfulness and certainty fidelity under both types of adversarial edits, but still missed some substitutions.","excerpt":"GPT-5-mini achieved the highest factual faithfulness and certainty fidelity under both adversarial edits.","url":"https://arxiv.org/abs/2611.10277","relationship":"Source-stated"},{"text":"For GPT-5-mini summaries, anatomical terms were not reflected in 23.7% of cases, diagnostic certainty in 22.3%, and descriptor terms in 21.2%.","excerpt":"substitutions of anatomical terms were not reflected in 23.7% of summaries, diagnostic certainty in 22.3%, and descriptor terms (e.g., \"substantial\") in 21.2%.","url":"https://arxiv.org/abs/2611.10277","relationship":"Source-stated"},{"text":"The authors conclude that adversarial evaluation exposes clinically meaningful failure modes in patient-friendly radiology summarization.","excerpt":"Adversarial evaluation exposes clinically meaningful failure modes, including missed anatomical modifiers, altered diagnostic certainty, and omitted severity descriptors.","url":"https://arxiv.org/abs/2611.10277","relationship":"Source-stated"},{"text":"The authors report that the automated judge's agreement with two radiologists was comparable to agreement between radiologists.","excerpt":"Judge-radiologist agreement (macro-F1: 0.55-0.56) was comparable to inter-radiologist agreement (0.53).","url":"https://arxiv.org/abs/2611.10277","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10277","checkedAt":"2026-11-12T11:34:08.625Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How often do the adversarial edits used in this evaluation reflect changes or omissions found in real radiology reports?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Adversarial Evaluation of Large Language Models for Patient-Friendly Radiology Report Summarization","summary":"The authors of this study evaluate whether large language models (LLMs) preserve clinical details when turning radiology reports into patient-friendly summaries, including when report wording is deliberately changed or information is removed. They tested six LLMs on CT and MRI reports, using two prompt styles and 18,000 evaluation pairs. Their results distinguish between kinds of edits: impression removals were handled more robustly than term substitutions. GPT-5-mini had the strongest scores for factual faithfulness and certainty fidelity, yet its summaries still missed changed anatomical terms, diagnostic certainty, and severity descriptors in some cases. The authors argue that adversarial evaluation can expose clinically meaningful failures. For patients, those omissions could matter because a summary may not reflect a detail or degree of certainty present in the report; the study evaluates summaries rather than patient outcomes. The authors also report that agreement between their automated judge and radiologists was comparable to agreement between two radiologists. The work was submitted to Radiology: Artificial Intelligence for possible publication; the evidence here is its arXiv abstract.","question":"How often do the adversarial edits used in this evaluation reflect changes or omissions found in real radiology reports?","claims":[{"text":"The authors evaluated six LLMs on adversarially modified CT and MRI reports, producing 18,000 evaluation pairs.","excerpt":"Six LLMs summarized original and adversarial impressions using 2 prompt styles, yielding 18,000 evaluation pairs.","support":"source-stated"},{"text":"Impression-removal edits were handled more robustly than term substitutions.","excerpt":"Impression-removal edits were handled more robustly than term substitutions.","support":"source-stated"},{"text":"GPT-5-mini achieved the highest factual faithfulness and certainty fidelity under both types of adversarial edits, but still missed some substitutions.","excerpt":"GPT-5-mini achieved the highest factual faithfulness and certainty fidelity under both adversarial edits.","support":"source-stated"},{"text":"For GPT-5-mini summaries, anatomical terms were not reflected in 23.7% of cases, diagnostic certainty in 22.3%, and descriptor terms in 21.2%.","excerpt":"substitutions of anatomical terms were not reflected in 23.7% of summaries, diagnostic certainty in 22.3%, and descriptor terms (e.g., \"substantial\") in 21.2%.","support":"source-stated"},{"text":"The authors conclude that adversarial evaluation exposes clinically meaningful failure modes in patient-friendly radiology summarization.","excerpt":"Adversarial evaluation exposes clinically meaningful failure modes, including missed anatomical modifiers, altered diagnostic certainty, and omitted severity descriptors.","support":"source-stated"},{"text":"The authors report that the automated judge's agreement with two radiologists was comparable to agreement between radiologists.","excerpt":"Judge-radiologist agreement (macro-F1: 0.55-0.56) was comparable to inter-radiologist agreement (0.53).","support":"source-stated"}],"limitations":["The evidence is the paper's arXiv abstract; it states that the work was submitted to Radiology: Artificial Intelligence for possible publication.","The evaluation uses adversarial variants of CT and MRI reports; the abstract does not establish how often those edits occur in routine reports.","The abstract reports evaluation scores and agreement, not effects on patients or clinical outcomes."],"mediaId":"b41c4efc28546bf6f73bcb59fba8b5f8e39fb8dcf71136e10f85ec0f880efcc0","correctionReason":null,"id":"d3185204082cbcdf3edf4e5d4ced394637504f42b703fc6e542f64d48bf548be","publicationId":"ed5f7324-99e6-4ae9-8069-4149c48aaf73","version":1,"sourceUrl":"https://arxiv.org/abs/2611.10277","sourceTitle":"Adversarial Evaluation of Large Language Models for Patient-Friendly Radiology Report Summarization","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-06","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-10T08:42:26.467Z","createdAt":"2026-11-12T11:03:52.633Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"b41c4efc28546bf6f73bcb59fba8b5f8e39fb8dcf71136e10f85ec0f880efcc0","kind":"article","url":"https://arxiv.org/abs/2611.10277","sourceUrl":"https://arxiv.org/abs/2611.10277","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T11:02:33.080Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10277","checkedAt":"2026-11-12T11:34:08.625Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"b41c4efc28546bf6f73bcb59fba8b5f8e39fb8dcf71136e10f85ec0f880efcc0","confidence":0.77,"provider":"typesafe","model":"jev-1.13.0","inputHash":"21cfe3810d239a7dbb051388272f1e3442d45ab65c6b05891c010e6111b370e5","decidedAt":"2026-11-12T11:02:33.839Z"},"provenance":{"textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observationId":"e8c5f5ec2f2213021b902fd6e06aa28d0ed8eb12d5e1bb97d7eddd47d968dd1d","retrievedAt":"2026-11-12T11:02:32.985Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"cfc9eca7-56cb-4f8f-80e8-51ed98653af6"},"reviewer":{"model":"gpt-6-luna","runId":"94a9e9d1-d7fa-4ebe-8e59-4aae73017072"}},"audit":{"status":"passed","reviewedAt":"2026-11-12T11:03:52.633Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Paschali, Magdalini","url":null,"kind":"unknown"},{"name":"Aali, Asad","url":null,"kind":"unknown"},{"name":"Koirala, Arogya","url":null,"kind":"unknown"},{"name":"Johnston, Andrew","url":null,"kind":"unknown"},{"name":"McNamara, Katherine","url":null,"kind":"unknown"},{"name":"Gaus, Richard","url":null,"kind":"unknown"},{"name":"Daneshjou, Roxana","url":null,"kind":"unknown"},{"name":"Larson, David B.","url":null,"kind":"unknown"},{"name":"Chaudhari, Akshay S.","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Paschali, Magdalini","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Paschali, Magdalini","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10277","locator":"head/meta[name=author|citation_author]"},"assertionId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:34:08.625Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","sourceUrl":"https://arxiv.org/abs/2611.10277","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z"}},{"actor":{"name":"Aali, Asad","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Aali, Asad","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10277","locator":"head/meta[name=author|citation_author]"},"assertionId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:34:08.625Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","sourceUrl":"https://arxiv.org/abs/2611.10277","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z"}},{"actor":{"name":"Koirala, Arogya","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Koirala, Arogya","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10277","locator":"head/meta[name=author|citation_author]"},"assertionId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:34:08.625Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","sourceUrl":"https://arxiv.org/abs/2611.10277","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z"}},{"actor":{"name":"Johnston, Andrew","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Johnston, Andrew","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10277","locator":"head/meta[name=author|citation_author]"},"assertionId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:34:08.625Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","sourceUrl":"https://arxiv.org/abs/2611.10277","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z"}},{"actor":{"name":"McNamara, Katherine","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"McNamara, Katherine","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10277","locator":"head/meta[name=author|citation_author]"},"assertionId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:34:08.625Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","sourceUrl":"https://arxiv.org/abs/2611.10277","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z"}},{"actor":{"name":"Gaus, Richard","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gaus, Richard","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10277","locator":"head/meta[name=author|citation_author]"},"assertionId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:34:08.625Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","sourceUrl":"https://arxiv.org/abs/2611.10277","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z"}},{"actor":{"name":"Daneshjou, Roxana","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Daneshjou, Roxana","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10277","locator":"head/meta[name=author|citation_author]"},"assertionId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:34:08.625Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","sourceUrl":"https://arxiv.org/abs/2611.10277","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z"}},{"actor":{"name":"Larson, David B.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Larson, David B.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10277","locator":"head/meta[name=author|citation_author]"},"assertionId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:34:08.625Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","sourceUrl":"https://arxiv.org/abs/2611.10277","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z"}},{"actor":{"name":"Chaudhari, Akshay S.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chaudhari, Akshay S.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10277","locator":"head/meta[name=author|citation_author]"},"assertionId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7:participation:8","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:34:08.625Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","sourceUrl":"https://arxiv.org/abs/2611.10277","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z"}}],"participationCoverage":{"total":9,"returned":9,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.10277","snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/d3185204082cbcdf3edf4e5d4ced394637504f42b703fc6e542f64d48bf548be/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-adversarial-evaluation-of-large-language-models-for-d3185204","formats":{"html":"/readings/arxiv-cs-ai-adversarial-evaluation-of-large-language-models-for-d3185204","markdown":"/readings/arxiv-cs-ai-adversarial-evaluation-of-large-language-models-for-d3185204.md","json":"/readings/arxiv-cs-ai-adversarial-evaluation-of-large-language-models-for-d3185204.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"d3185204082cbcdf3edf4e5d4ced394637504f42b703fc6e542f64d48bf548be","revision":"ed5f7324-99e6-4ae9-8069-4149c48aaf73","revisionKind":"publication-id","title":"arXiv cs.AI: Adversarial Evaluation of Large Language Models for Patient-Friendly Radiology Report Summarization","url":"https://escapemeridian.com/readings/arxiv-cs-ai-adversarial-evaluation-of-large-language-models-for-d3185204"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.10277","revision":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","revisionKind":"sha256-source-text","title":"Adversarial Evaluation of Large Language Models for Patient-Friendly Radiology Report Summarization","url":"https://arxiv.org/abs/2611.10277"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"d3185204082cbcdf3edf4e5d4ced394637504f42b703fc6e542f64d48bf548be","title":"arXiv cs.AI: Adversarial Evaluation of Large Language Models for Patient-Friendly Radiology Report Summarization","argument":"The authors of this study evaluate whether large language models (LLMs) preserve clinical details when turning radiology reports into patient-friendly summaries, including when report wording is deliberately changed or information is removed. They tested six LLMs on CT and MRI reports, using two prompt styles and 18,000 evaluation pairs. Their results distinguish between kinds of edits: impression removals were handled more robustly than term substitutions. GPT-5-mini had the strongest scores for factual faithfulness and certainty fidelity, yet its summaries still missed changed anatomical terms, diagnostic certainty, and severity descriptors in some cases. The authors argue that adversarial evaluation can expose clinically meaningful failures. For patients, those omissions could matter because a summary may not reflect a detail or degree of certainty present in the report; the study evaluates summaries rather than patient outcomes. The authors also report that agreement between their automated judge and radiologists was comparable to agreement between two radiologists. The work was submitted to Radiology: Artificial Intelligence for possible publication; the evidence here is its arXiv abstract.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence is the paper's arXiv abstract; it states that the work was submitted to Radiology: Artificial Intelligence for possible publication. The evaluation uses adversarial variants of CT and MRI reports; the abstract does not establish how often those edits occur in routine reports. The abstract reports evaluation scores and agreement, not effects on patients or clinical outcomes.","speaker":{"name":"Paschali, Magdalini, Aali, Asad, Koirala, Arogya, Johnston, Andrew, McNamara, Katherine, Gaus, Richard, Daneshjou, Roxana, Larson, David B., Chaudhari, Akshay S.","roleAtSource":"Article author"},"source":{"title":"Adversarial Evaluation of Large Language Models for Patient-Friendly Radiology Report Summarization","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.10277","publishedAt":"2026-11-06","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors evaluated six LLMs on adversarially modified CT and MRI reports, producing 18,000 evaluation pairs.","excerpt":"Six LLMs summarized original and adversarial impressions using 2 prompt styles, yielding 18,000 evaluation pairs.","url":"https://arxiv.org/abs/2611.10277","relationship":"Source-stated"},{"text":"Impression-removal edits were handled more robustly than term substitutions.","excerpt":"Impression-removal edits were handled more robustly than term substitutions.","url":"https://arxiv.org/abs/2611.10277","relationship":"Source-stated"},{"text":"GPT-5-mini achieved the highest factual faithfulness and certainty fidelity under both types of adversarial edits, but still missed some substitutions.","excerpt":"GPT-5-mini achieved the highest factual faithfulness and certainty fidelity under both adversarial edits.","url":"https://arxiv.org/abs/2611.10277","relationship":"Source-stated"},{"text":"For GPT-5-mini summaries, anatomical terms were not reflected in 23.7% of cases, diagnostic certainty in 22.3%, and descriptor terms in 21.2%.","excerpt":"substitutions of anatomical terms were not reflected in 23.7% of summaries, diagnostic certainty in 22.3%, and descriptor terms (e.g., \"substantial\") in 21.2%.","url":"https://arxiv.org/abs/2611.10277","relationship":"Source-stated"},{"text":"The authors conclude that adversarial evaluation exposes clinically meaningful failure modes in patient-friendly radiology summarization.","excerpt":"Adversarial evaluation exposes clinically meaningful failure modes, including missed anatomical modifiers, altered diagnostic certainty, and omitted severity descriptors.","url":"https://arxiv.org/abs/2611.10277","relationship":"Source-stated"},{"text":"The authors report that the automated judge's agreement with two radiologists was comparable to agreement between radiologists.","excerpt":"Judge-radiologist agreement (macro-F1: 0.55-0.56) was comparable to inter-radiologist agreement (0.53).","url":"https://arxiv.org/abs/2611.10277","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10277","checkedAt":"2026-11-12T11:34:08.625Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How often do the adversarial edits used in this evaluation reflect changes or omissions found in real radiology reports?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Adversarial Evaluation of Large Language Models for Patient-Friendly Radiology Report Summarization","summary":"The authors of this study evaluate whether large language models (LLMs) preserve clinical details when turning radiology reports into patient-friendly summaries, including when report wording is deliberately changed or information is removed. They tested six LLMs on CT and MRI reports, using two prompt styles and 18,000 evaluation pairs. Their results distinguish between kinds of edits: impression removals were handled more robustly than term substitutions. GPT-5-mini had the strongest scores for factual faithfulness and certainty fidelity, yet its summaries still missed changed anatomical terms, diagnostic certainty, and severity descriptors in some cases. The authors argue that adversarial evaluation can expose clinically meaningful failures. For patients, those omissions could matter because a summary may not reflect a detail or degree of certainty present in the report; the study evaluates summaries rather than patient outcomes. The authors also report that agreement between their automated judge and radiologists was comparable to agreement between two radiologists. The work was submitted to Radiology: Artificial Intelligence for possible publication; the evidence here is its arXiv abstract.","question":"How often do the adversarial edits used in this evaluation reflect changes or omissions found in real radiology reports?","claims":[{"text":"The authors evaluated six LLMs on adversarially modified CT and MRI reports, producing 18,000 evaluation pairs.","excerpt":"Six LLMs summarized original and adversarial impressions using 2 prompt styles, yielding 18,000 evaluation pairs.","support":"source-stated"},{"text":"Impression-removal edits were handled more robustly than term substitutions.","excerpt":"Impression-removal edits were handled more robustly than term substitutions.","support":"source-stated"},{"text":"GPT-5-mini achieved the highest factual faithfulness and certainty fidelity under both types of adversarial edits, but still missed some substitutions.","excerpt":"GPT-5-mini achieved the highest factual faithfulness and certainty fidelity under both adversarial edits.","support":"source-stated"},{"text":"For GPT-5-mini summaries, anatomical terms were not reflected in 23.7% of cases, diagnostic certainty in 22.3%, and descriptor terms in 21.2%.","excerpt":"substitutions of anatomical terms were not reflected in 23.7% of summaries, diagnostic certainty in 22.3%, and descriptor terms (e.g., \"substantial\") in 21.2%.","support":"source-stated"},{"text":"The authors conclude that adversarial evaluation exposes clinically meaningful failure modes in patient-friendly radiology summarization.","excerpt":"Adversarial evaluation exposes clinically meaningful failure modes, including missed anatomical modifiers, altered diagnostic certainty, and omitted severity descriptors.","support":"source-stated"},{"text":"The authors report that the automated judge's agreement with two radiologists was comparable to agreement between radiologists.","excerpt":"Judge-radiologist agreement (macro-F1: 0.55-0.56) was comparable to inter-radiologist agreement (0.53).","support":"source-stated"}],"limitations":["The evidence is the paper's arXiv abstract; it states that the work was submitted to Radiology: Artificial Intelligence for possible publication.","The evaluation uses adversarial variants of CT and MRI reports; the abstract does not establish how often those edits occur in routine reports.","The abstract reports evaluation scores and agreement, not effects on patients or clinical outcomes."],"mediaId":"b41c4efc28546bf6f73bcb59fba8b5f8e39fb8dcf71136e10f85ec0f880efcc0","correctionReason":null,"id":"d3185204082cbcdf3edf4e5d4ced394637504f42b703fc6e542f64d48bf548be","publicationId":"ed5f7324-99e6-4ae9-8069-4149c48aaf73","version":1,"sourceUrl":"https://arxiv.org/abs/2611.10277","sourceTitle":"Adversarial Evaluation of Large Language Models for Patient-Friendly Radiology Report Summarization","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-06","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-10T08:42:26.467Z","createdAt":"2026-11-12T11:03:52.633Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"b41c4efc28546bf6f73bcb59fba8b5f8e39fb8dcf71136e10f85ec0f880efcc0","kind":"article","url":"https://arxiv.org/abs/2611.10277","sourceUrl":"https://arxiv.org/abs/2611.10277","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T11:02:33.080Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10277","checkedAt":"2026-11-12T11:34:08.625Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"b41c4efc28546bf6f73bcb59fba8b5f8e39fb8dcf71136e10f85ec0f880efcc0","confidence":0.77,"provider":"typesafe","model":"jev-1.13.0","inputHash":"21cfe3810d239a7dbb051388272f1e3442d45ab65c6b05891c010e6111b370e5","decidedAt":"2026-11-12T11:02:33.839Z"},"provenance":{"textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observationId":"e8c5f5ec2f2213021b902fd6e06aa28d0ed8eb12d5e1bb97d7eddd47d968dd1d","retrievedAt":"2026-11-12T11:02:32.985Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"cfc9eca7-56cb-4f8f-80e8-51ed98653af6"},"reviewer":{"model":"gpt-6-luna","runId":"94a9e9d1-d7fa-4ebe-8e59-4aae73017072"}},"audit":{"status":"passed","reviewedAt":"2026-11-12T11:03:52.633Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Paschali, Magdalini","url":null,"kind":"unknown"},{"name":"Aali, Asad","url":null,"kind":"unknown"},{"name":"Koirala, Arogya","url":null,"kind":"unknown"},{"name":"Johnston, Andrew","url":null,"kind":"unknown"},{"name":"McNamara, Katherine","url":null,"kind":"unknown"},{"name":"Gaus, Richard","url":null,"kind":"unknown"},{"name":"Daneshjou, Roxana","url":null,"kind":"unknown"},{"name":"Larson, David B.","url":null,"kind":"unknown"},{"name":"Chaudhari, Akshay S.","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Paschali, Magdalini","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Paschali, Magdalini","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10277","locator":"head/meta[name=author|citation_author]"},"assertionId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:34:08.625Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","sourceUrl":"https://arxiv.org/abs/2611.10277","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z"}},{"actor":{"name":"Aali, Asad","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Aali, Asad","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10277","locator":"head/meta[name=author|citation_author]"},"assertionId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:34:08.625Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","sourceUrl":"https://arxiv.org/abs/2611.10277","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z"}},{"actor":{"name":"Koirala, Arogya","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Koirala, Arogya","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10277","locator":"head/meta[name=author|citation_author]"},"assertionId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:34:08.625Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","sourceUrl":"https://arxiv.org/abs/2611.10277","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z"}},{"actor":{"name":"Johnston, Andrew","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Johnston, Andrew","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10277","locator":"head/meta[name=author|citation_author]"},"assertionId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:34:08.625Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","sourceUrl":"https://arxiv.org/abs/2611.10277","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z"}},{"actor":{"name":"McNamara, Katherine","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"McNamara, Katherine","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10277","locator":"head/meta[name=author|citation_author]"},"assertionId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:34:08.625Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","sourceUrl":"https://arxiv.org/abs/2611.10277","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z"}},{"actor":{"name":"Gaus, Richard","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gaus, Richard","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10277","locator":"head/meta[name=author|citation_author]"},"assertionId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:34:08.625Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","sourceUrl":"https://arxiv.org/abs/2611.10277","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z"}},{"actor":{"name":"Daneshjou, Roxana","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Daneshjou, Roxana","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10277","locator":"head/meta[name=author|citation_author]"},"assertionId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:34:08.625Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","sourceUrl":"https://arxiv.org/abs/2611.10277","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z"}},{"actor":{"name":"Larson, David B.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Larson, David B.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10277","locator":"head/meta[name=author|citation_author]"},"assertionId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:34:08.625Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","sourceUrl":"https://arxiv.org/abs/2611.10277","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z"}},{"actor":{"name":"Chaudhari, Akshay S.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chaudhari, Akshay S.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10277","locator":"head/meta[name=author|citation_author]"},"assertionId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7:participation:8","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:34:08.625Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","sourceUrl":"https://arxiv.org/abs/2611.10277","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z"}}],"participationCoverage":{"total":9,"returned":9,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.10277","snapshotId":"305a7cb685c4b61f7e3ac6dfcc23570b84a1b4a0c394ab90a0ba19f57a7111a7","contentSha256":"e7a3e12419ebcbf0ff69325a36fdb5f0a56badf3d49c726d38793c70f628f8f9","textHash":"70859585b6ce2aecb4c0efa5550ea93e0f6da94c5d0cd3b21defdbeeaa5561b9","observedAt":"2026-11-12T11:34:08.625Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/d3185204082cbcdf3edf4e5d4ced394637504f42b703fc6e542f64d48bf548be/source"}}}],"audits":[{"publicationId":"ed5f7324-99e6-4ae9-8069-4149c48aaf73","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the framework, results and conclusion to the paper's authors, and correctly names arXiv cs.AI as the publisher. Model results are reported as findings, not presented as EM's own claims.","excerpt":"Authors:Magdalini Paschali, Asad Aali, Arogya Koirala, Andrew Johnston, Katherine McNamara, Richard Gaus, Roxana Daneshjou, David B. Larson, Akshay S. Chaudhari"},{"dimension":"summary","status":"pass","explanation":"The reading accurately covers the setup (six LLMs, two prompt styles, 18,000 pairs), the comparison of edit types, GPT-5-mini's lead and its remaining misses, judge agreement and submission status. 'Adversarial' is explained as deliberately changing wording or removing information. 'Impression' (the conclusion section of a radiology report) is not explained but does not block understanding; it is a candidate for a term-library tooltip. The omitted hallucination-rate comparison does not distort the reading. The question is a genuine open issue.","excerpt":"Impression-removal edits were handled more robustly than term substitutions."},{"dimension":"date","status":"pass","explanation":"originalPublished 2026-11-06 matches the v1 submission date on arXiv. The feed date of Nov 10 is correctly treated as later.","excerpt":"[Submitted on 6 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"All claims match the abstract. The abstract attributes the 23.7/22.3/21.2% figures to 'the highest-scoring summarizer'. The reading assigns them to GPT-5-mini, which fits the abstract's statement that GPT-5-mini scored highest. The patient-impact sentence is hedged ('could matter') and explicitly separates summary evaluation from patient outcomes. It is grounded in the authors' 'clinically meaningful' conclusion. The limitations are accurate.","excerpt":"Even for the highest-scoring summarizer, substitutions of anatomical terms were not reflected in 23.7% of summaries, diagnostic certainty in 22.3%, and descriptor terms (e.g., \"substantial\") in 21.2%."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.","excerpt":null}],"createdAt":"2026-11-12T11:03:52.633Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}