{"id":"1d4f1ea149682ee84258c32e6044675e3a776fdf7b2a1c1e8d0c50f545f9beb6","title":"arXiv cs.AI paper tests multimodal medical reasoning with synthetic images","argument":"The authors of Trick-or-Treat introduce an adversarial benchmark that pairs real chest radiographs with synthetic clinical photographs. The paired images either support the same diagnosis or deliberately conflict, while synthetic photographs let the researchers vary clinical findings without changing the person’s features or photographic setting. In tests of three frontier models, the authors report that models hallucinated clinically significant photograph findings in 45-57.5% of mismatched pairs, compared with 25-37.5% when the same photographs appeared alone. They say models often ignored the mismatch, misreading a finding such as acne as crusted skin lesions to support a diagnosis suggested by the radiograph. The authors present the benchmark as a way to expose this error pattern and as a proof of concept for using synthetic images in adversarial multimodal evaluation. The results concern this benchmark and the three models investigated; the abstract does not identify those models or give model-by-model results.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported rates are from an investigation of three frontier models; the supplied abstract does not name them or provide model-by-model results. The supplied abstract reports benchmark findings, not how these results translate to clinical practice.","speaker":{"name":"Banerjee, Oishi, Lee, Eujin, Kim, Sung Eun, Willauer, Alexandra N., Zhou, Christine Y., Yang, Zhizhou, Rajpurkar, Pranav","roleAtSource":"Article author"},"source":{"title":"Trick-or-Treat: Adversarial Evaluation of Multimodal Medical Reasoning with Synthetic Images","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.06631","publishedAt":"2026-11-04","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce Trick-or-Treat, an adversarial benchmark pairing real chest radiographs with synthetic clinical photographs to test cross-modal reasoning.","excerpt":"we introduce Trick-or-Treat, an adversarial multimodal medical benchmark that pairs real chest radiographs with synthetic clinical photographs","url":"https://arxiv.org/abs/2611.06631","relationship":"Source-stated"},{"text":"The benchmark compares matched pairs, where both images suggest the same disease, with mismatched pairs, where the photograph suggests a different finding.","excerpt":"In matched pairs, the chest radiograph and photograph both contain findings suggesting the same disease","url":"https://arxiv.org/abs/2611.06631","relationship":"Source-stated"},{"text":"For mismatched pairs, the authors report hallucinated clinically significant photograph findings in 45-57.5% of cases, versus 25-37.5% when the same photographs were shown alone.","excerpt":"models hallucinated clinically significant photograph findings in 45-57.5% of mismatched pairs, up from 25-37.5% for the same photographs shown alone.","url":"https://arxiv.org/abs/2611.06631","relationship":"Source-stated"},{"text":"The authors report that models tended to disregard mismatches and infer photograph findings that fit the radiograph, such as reading acne as crusted skin lesions.","excerpt":"models tended to ignore mismatches and instead hallucinated findings that better matched the radiograph (e.g. misreading acne as crusted skin lesions to justify diagnosing chickenpox).","url":"https://arxiv.org/abs/2611.06631","relationship":"Source-stated"},{"text":"The authors describe the work as a proof of concept for synthetic images in adversarial multimodal evaluation.","excerpt":"provides a proof of concept for the use of synthetic images in adversarial multimodal evaluation.","url":"https://arxiv.org/abs/2611.06631","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.06631","checkedAt":"2026-11-09T12:56:07.244Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How do the reported hallucination rates vary across the three models and the conditions tested?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI paper tests multimodal medical reasoning with synthetic images","summary":"The authors of Trick-or-Treat introduce an adversarial benchmark that pairs real chest radiographs with synthetic clinical photographs. The paired images either support the same diagnosis or deliberately conflict, while synthetic photographs let the researchers vary clinical findings without changing the person’s features or photographic setting. In tests of three frontier models, the authors report that models hallucinated clinically significant photograph findings in 45-57.5% of mismatched pairs, compared with 25-37.5% when the same photographs appeared alone. They say models often ignored the mismatch, misreading a finding such as acne as crusted skin lesions to support a diagnosis suggested by the radiograph. The authors present the benchmark as a way to expose this error pattern and as a proof of concept for using synthetic images in adversarial multimodal evaluation. The results concern this benchmark and the three models investigated; the abstract does not identify those models or give model-by-model results.","question":"How do the reported hallucination rates vary across the three models and the conditions tested?","claims":[{"text":"The authors introduce Trick-or-Treat, an adversarial benchmark pairing real chest radiographs with synthetic clinical photographs to test cross-modal reasoning.","excerpt":"we introduce Trick-or-Treat, an adversarial multimodal medical benchmark that pairs real chest radiographs with synthetic clinical photographs","support":"source-stated"},{"text":"The benchmark compares matched pairs, where both images suggest the same disease, with mismatched pairs, where the photograph suggests a different finding.","excerpt":"In matched pairs, the chest radiograph and photograph both contain findings suggesting the same disease","support":"source-stated"},{"text":"For mismatched pairs, the authors report hallucinated clinically significant photograph findings in 45-57.5% of cases, versus 25-37.5% when the same photographs were shown alone.","excerpt":"models hallucinated clinically significant photograph findings in 45-57.5% of mismatched pairs, up from 25-37.5% for the same photographs shown alone.","support":"source-stated"},{"text":"The authors report that models tended to disregard mismatches and infer photograph findings that fit the radiograph, such as reading acne as crusted skin lesions.","excerpt":"models tended to ignore mismatches and instead hallucinated findings that better matched the radiograph (e.g. misreading acne as crusted skin lesions to justify diagnosing chickenpox).","support":"source-stated"},{"text":"The authors describe the work as a proof of concept for synthetic images in adversarial multimodal evaluation.","excerpt":"provides a proof of concept for the use of synthetic images in adversarial multimodal evaluation.","support":"source-stated"}],"limitations":["The reported rates are from an investigation of three frontier models; the supplied abstract does not name them or provide model-by-model results.","The supplied abstract reports benchmark findings, not how these results translate to clinical practice."],"mediaId":"4141a89707cad62f403f42df62fbf4c91e0fa6f3d30d54990c6c3638bba76b42","correctionReason":null,"id":"1d4f1ea149682ee84258c32e6044675e3a776fdf7b2a1c1e8d0c50f545f9beb6","publicationId":"30b1e63e-409b-4667-87a3-c1a3be59763c","version":1,"sourceUrl":"https://arxiv.org/abs/2611.06631","sourceTitle":"Trick-or-Treat: Adversarial Evaluation of Multimodal Medical Reasoning with Synthetic Images","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-04","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-05T05:06:37.056Z","createdAt":"2026-11-09T12:24:52.014Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"4141a89707cad62f403f42df62fbf4c91e0fa6f3d30d54990c6c3638bba76b42","kind":"article","url":"https://arxiv.org/abs/2611.06631","sourceUrl":"https://arxiv.org/abs/2611.06631","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-09T12:24:10.856Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.06631","checkedAt":"2026-11-09T12:56:07.244Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"4141a89707cad62f403f42df62fbf4c91e0fa6f3d30d54990c6c3638bba76b42","confidence":0.8700000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"86c0a2042d7637531d957d50c18f8ff16e6ae1e789e1eccde12d642ea19b0e2c","decidedAt":"2026-11-09T12:24:11.913Z"},"provenance":{"textHash":"069473adfe0c403bf483b2a206ee3666c8a15f4b37e6c15f6195137826b46dfb","observationId":"b1c6851c10dca7072a81597416129dfdbb8c732cb541511f4833d88a57a995e5","retrievedAt":"2026-11-09T12:24:10.677Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"fb347db4-1b8c-4754-8dc0-1f088f73a8aa"},"reviewer":{"model":"gpt-6-luna","runId":"d623bf14-e239-47ea-8e03-45d6314ba56a"}},"audit":{"status":"passed","reviewedAt":"2026-11-09T12:24:52.014Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Banerjee, Oishi","url":null,"kind":"unknown"},{"name":"Lee, Eujin","url":null,"kind":"unknown"},{"name":"Kim, Sung Eun","url":null,"kind":"unknown"},{"name":"Willauer, Alexandra N.","url":null,"kind":"unknown"},{"name":"Zhou, Christine Y.","url":null,"kind":"unknown"},{"name":"Yang, Zhizhou","url":null,"kind":"unknown"},{"name":"Rajpurkar, Pranav","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Banerjee, Oishi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Banerjee, Oishi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06631","locator":"head/meta[name=author|citation_author]"},"assertionId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T12:56:07.244Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c","sourceUrl":"https://arxiv.org/abs/2611.06631","contentSha256":"ed565a5f5a0e942ee2160dbb8a24f974465b4127365bb5b2d2e05daee6ae80a0","textHash":"069473adfe0c403bf483b2a206ee3666c8a15f4b37e6c15f6195137826b46dfb","observedAt":"2026-11-09T12:56:07.244Z"}},{"actor":{"name":"Lee, Eujin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Lee, Eujin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06631","locator":"head/meta[name=author|citation_author]"},"assertionId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T12:56:07.244Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c","sourceUrl":"https://arxiv.org/abs/2611.06631","contentSha256":"ed565a5f5a0e942ee2160dbb8a24f974465b4127365bb5b2d2e05daee6ae80a0","textHash":"069473adfe0c403bf483b2a206ee3666c8a15f4b37e6c15f6195137826b46dfb","observedAt":"2026-11-09T12:56:07.244Z"}},{"actor":{"name":"Kim, Sung Eun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Kim, Sung Eun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06631","locator":"head/meta[name=author|citation_author]"},"assertionId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T12:56:07.244Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c","sourceUrl":"https://arxiv.org/abs/2611.06631","contentSha256":"ed565a5f5a0e942ee2160dbb8a24f974465b4127365bb5b2d2e05daee6ae80a0","textHash":"069473adfe0c403bf483b2a206ee3666c8a15f4b37e6c15f6195137826b46dfb","observedAt":"2026-11-09T12:56:07.244Z"}},{"actor":{"name":"Willauer, Alexandra N.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Willauer, Alexandra N.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06631","locator":"head/meta[name=author|citation_author]"},"assertionId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T12:56:07.244Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c","sourceUrl":"https://arxiv.org/abs/2611.06631","contentSha256":"ed565a5f5a0e942ee2160dbb8a24f974465b4127365bb5b2d2e05daee6ae80a0","textHash":"069473adfe0c403bf483b2a206ee3666c8a15f4b37e6c15f6195137826b46dfb","observedAt":"2026-11-09T12:56:07.244Z"}},{"actor":{"name":"Zhou, Christine Y.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhou, Christine Y.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06631","locator":"head/meta[name=author|citation_author]"},"assertionId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T12:56:07.244Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c","sourceUrl":"https://arxiv.org/abs/2611.06631","contentSha256":"ed565a5f5a0e942ee2160dbb8a24f974465b4127365bb5b2d2e05daee6ae80a0","textHash":"069473adfe0c403bf483b2a206ee3666c8a15f4b37e6c15f6195137826b46dfb","observedAt":"2026-11-09T12:56:07.244Z"}},{"actor":{"name":"Yang, Zhizhou","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yang, Zhizhou","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06631","locator":"head/meta[name=author|citation_author]"},"assertionId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T12:56:07.244Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c","sourceUrl":"https://arxiv.org/abs/2611.06631","contentSha256":"ed565a5f5a0e942ee2160dbb8a24f974465b4127365bb5b2d2e05daee6ae80a0","textHash":"069473adfe0c403bf483b2a206ee3666c8a15f4b37e6c15f6195137826b46dfb","observedAt":"2026-11-09T12:56:07.244Z"}},{"actor":{"name":"Rajpurkar, Pranav","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Rajpurkar, Pranav","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06631","locator":"head/meta[name=author|citation_author]"},"assertionId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T12:56:07.244Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c","sourceUrl":"https://arxiv.org/abs/2611.06631","contentSha256":"ed565a5f5a0e942ee2160dbb8a24f974465b4127365bb5b2d2e05daee6ae80a0","textHash":"069473adfe0c403bf483b2a206ee3666c8a15f4b37e6c15f6195137826b46dfb","observedAt":"2026-11-09T12:56:07.244Z"}}],"participationCoverage":{"total":7,"returned":7,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.06631","snapshotId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c","contentSha256":"ed565a5f5a0e942ee2160dbb8a24f974465b4127365bb5b2d2e05daee6ae80a0","textHash":"069473adfe0c403bf483b2a206ee3666c8a15f4b37e6c15f6195137826b46dfb","observedAt":"2026-11-09T12:56:07.244Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/1d4f1ea149682ee84258c32e6044675e3a776fdf7b2a1c1e8d0c50f545f9beb6/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-paper-tests-multimodal-medical-reasoning-with-1d4f1ea1","formats":{"html":"/readings/arxiv-cs-ai-paper-tests-multimodal-medical-reasoning-with-1d4f1ea1","markdown":"/readings/arxiv-cs-ai-paper-tests-multimodal-medical-reasoning-with-1d4f1ea1.md","json":"/readings/arxiv-cs-ai-paper-tests-multimodal-medical-reasoning-with-1d4f1ea1.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-multimodal-model","name":"Multimodal model","explanation":"A model that works with more than one kind of input or output, such as text, images and audio.","referenceUrl":"https://developers.google.com/machine-learning/glossary#multimodal-model"},{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-frontier-model","name":"Frontier model","explanation":"One of the most capable AI models available at a given time; the set shifts as models improve.","referenceUrl":"https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper"},{"id":"term-hallucination","name":"Hallucination","explanation":"When an AI system states something false or made up as if it were true.","referenceUrl":"https://developers.google.com/machine-learning/glossary#hallucination"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"1d4f1ea149682ee84258c32e6044675e3a776fdf7b2a1c1e8d0c50f545f9beb6","revision":"30b1e63e-409b-4667-87a3-c1a3be59763c","revisionKind":"publication-id","title":"arXiv cs.AI paper tests multimodal medical reasoning with synthetic images","url":"https://escapemeridian.com/readings/arxiv-cs-ai-paper-tests-multimodal-medical-reasoning-with-1d4f1ea1"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.06631","revision":"069473adfe0c403bf483b2a206ee3666c8a15f4b37e6c15f6195137826b46dfb","revisionKind":"sha256-source-text","title":"Trick-or-Treat: Adversarial Evaluation of Multimodal Medical Reasoning with Synthetic Images","url":"https://arxiv.org/abs/2611.06631"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"1d4f1ea149682ee84258c32e6044675e3a776fdf7b2a1c1e8d0c50f545f9beb6","title":"arXiv cs.AI paper tests multimodal medical reasoning with synthetic images","argument":"The authors of Trick-or-Treat introduce an adversarial benchmark that pairs real chest radiographs with synthetic clinical photographs. The paired images either support the same diagnosis or deliberately conflict, while synthetic photographs let the researchers vary clinical findings without changing the person’s features or photographic setting. In tests of three frontier models, the authors report that models hallucinated clinically significant photograph findings in 45-57.5% of mismatched pairs, compared with 25-37.5% when the same photographs appeared alone. They say models often ignored the mismatch, misreading a finding such as acne as crusted skin lesions to support a diagnosis suggested by the radiograph. The authors present the benchmark as a way to expose this error pattern and as a proof of concept for using synthetic images in adversarial multimodal evaluation. The results concern this benchmark and the three models investigated; the abstract does not identify those models or give model-by-model results.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported rates are from an investigation of three frontier models; the supplied abstract does not name them or provide model-by-model results. The supplied abstract reports benchmark findings, not how these results translate to clinical practice.","speaker":{"name":"Banerjee, Oishi, Lee, Eujin, Kim, Sung Eun, Willauer, Alexandra N., Zhou, Christine Y., Yang, Zhizhou, Rajpurkar, Pranav","roleAtSource":"Article author"},"source":{"title":"Trick-or-Treat: Adversarial Evaluation of Multimodal Medical Reasoning with Synthetic Images","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.06631","publishedAt":"2026-11-04","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce Trick-or-Treat, an adversarial benchmark pairing real chest radiographs with synthetic clinical photographs to test cross-modal reasoning.","excerpt":"we introduce Trick-or-Treat, an adversarial multimodal medical benchmark that pairs real chest radiographs with synthetic clinical photographs","url":"https://arxiv.org/abs/2611.06631","relationship":"Source-stated"},{"text":"The benchmark compares matched pairs, where both images suggest the same disease, with mismatched pairs, where the photograph suggests a different finding.","excerpt":"In matched pairs, the chest radiograph and photograph both contain findings suggesting the same disease","url":"https://arxiv.org/abs/2611.06631","relationship":"Source-stated"},{"text":"For mismatched pairs, the authors report hallucinated clinically significant photograph findings in 45-57.5% of cases, versus 25-37.5% when the same photographs were shown alone.","excerpt":"models hallucinated clinically significant photograph findings in 45-57.5% of mismatched pairs, up from 25-37.5% for the same photographs shown alone.","url":"https://arxiv.org/abs/2611.06631","relationship":"Source-stated"},{"text":"The authors report that models tended to disregard mismatches and infer photograph findings that fit the radiograph, such as reading acne as crusted skin lesions.","excerpt":"models tended to ignore mismatches and instead hallucinated findings that better matched the radiograph (e.g. misreading acne as crusted skin lesions to justify diagnosing chickenpox).","url":"https://arxiv.org/abs/2611.06631","relationship":"Source-stated"},{"text":"The authors describe the work as a proof of concept for synthetic images in adversarial multimodal evaluation.","excerpt":"provides a proof of concept for the use of synthetic images in adversarial multimodal evaluation.","url":"https://arxiv.org/abs/2611.06631","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.06631","checkedAt":"2026-11-09T12:56:07.244Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How do the reported hallucination rates vary across the three models and the conditions tested?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI paper tests multimodal medical reasoning with synthetic images","summary":"The authors of Trick-or-Treat introduce an adversarial benchmark that pairs real chest radiographs with synthetic clinical photographs. The paired images either support the same diagnosis or deliberately conflict, while synthetic photographs let the researchers vary clinical findings without changing the person’s features or photographic setting. In tests of three frontier models, the authors report that models hallucinated clinically significant photograph findings in 45-57.5% of mismatched pairs, compared with 25-37.5% when the same photographs appeared alone. They say models often ignored the mismatch, misreading a finding such as acne as crusted skin lesions to support a diagnosis suggested by the radiograph. The authors present the benchmark as a way to expose this error pattern and as a proof of concept for using synthetic images in adversarial multimodal evaluation. The results concern this benchmark and the three models investigated; the abstract does not identify those models or give model-by-model results.","question":"How do the reported hallucination rates vary across the three models and the conditions tested?","claims":[{"text":"The authors introduce Trick-or-Treat, an adversarial benchmark pairing real chest radiographs with synthetic clinical photographs to test cross-modal reasoning.","excerpt":"we introduce Trick-or-Treat, an adversarial multimodal medical benchmark that pairs real chest radiographs with synthetic clinical photographs","support":"source-stated"},{"text":"The benchmark compares matched pairs, where both images suggest the same disease, with mismatched pairs, where the photograph suggests a different finding.","excerpt":"In matched pairs, the chest radiograph and photograph both contain findings suggesting the same disease","support":"source-stated"},{"text":"For mismatched pairs, the authors report hallucinated clinically significant photograph findings in 45-57.5% of cases, versus 25-37.5% when the same photographs were shown alone.","excerpt":"models hallucinated clinically significant photograph findings in 45-57.5% of mismatched pairs, up from 25-37.5% for the same photographs shown alone.","support":"source-stated"},{"text":"The authors report that models tended to disregard mismatches and infer photograph findings that fit the radiograph, such as reading acne as crusted skin lesions.","excerpt":"models tended to ignore mismatches and instead hallucinated findings that better matched the radiograph (e.g. misreading acne as crusted skin lesions to justify diagnosing chickenpox).","support":"source-stated"},{"text":"The authors describe the work as a proof of concept for synthetic images in adversarial multimodal evaluation.","excerpt":"provides a proof of concept for the use of synthetic images in adversarial multimodal evaluation.","support":"source-stated"}],"limitations":["The reported rates are from an investigation of three frontier models; the supplied abstract does not name them or provide model-by-model results.","The supplied abstract reports benchmark findings, not how these results translate to clinical practice."],"mediaId":"4141a89707cad62f403f42df62fbf4c91e0fa6f3d30d54990c6c3638bba76b42","correctionReason":null,"id":"1d4f1ea149682ee84258c32e6044675e3a776fdf7b2a1c1e8d0c50f545f9beb6","publicationId":"30b1e63e-409b-4667-87a3-c1a3be59763c","version":1,"sourceUrl":"https://arxiv.org/abs/2611.06631","sourceTitle":"Trick-or-Treat: Adversarial Evaluation of Multimodal Medical Reasoning with Synthetic Images","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-04","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-05T05:06:37.056Z","createdAt":"2026-11-09T12:24:52.014Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"4141a89707cad62f403f42df62fbf4c91e0fa6f3d30d54990c6c3638bba76b42","kind":"article","url":"https://arxiv.org/abs/2611.06631","sourceUrl":"https://arxiv.org/abs/2611.06631","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-09T12:24:10.856Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.06631","checkedAt":"2026-11-09T12:56:07.244Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"4141a89707cad62f403f42df62fbf4c91e0fa6f3d30d54990c6c3638bba76b42","confidence":0.8700000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"86c0a2042d7637531d957d50c18f8ff16e6ae1e789e1eccde12d642ea19b0e2c","decidedAt":"2026-11-09T12:24:11.913Z"},"provenance":{"textHash":"069473adfe0c403bf483b2a206ee3666c8a15f4b37e6c15f6195137826b46dfb","observationId":"b1c6851c10dca7072a81597416129dfdbb8c732cb541511f4833d88a57a995e5","retrievedAt":"2026-11-09T12:24:10.677Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"fb347db4-1b8c-4754-8dc0-1f088f73a8aa"},"reviewer":{"model":"gpt-6-luna","runId":"d623bf14-e239-47ea-8e03-45d6314ba56a"}},"audit":{"status":"passed","reviewedAt":"2026-11-09T12:24:52.014Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Banerjee, Oishi","url":null,"kind":"unknown"},{"name":"Lee, Eujin","url":null,"kind":"unknown"},{"name":"Kim, Sung Eun","url":null,"kind":"unknown"},{"name":"Willauer, Alexandra N.","url":null,"kind":"unknown"},{"name":"Zhou, Christine Y.","url":null,"kind":"unknown"},{"name":"Yang, Zhizhou","url":null,"kind":"unknown"},{"name":"Rajpurkar, Pranav","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Banerjee, Oishi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Banerjee, Oishi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06631","locator":"head/meta[name=author|citation_author]"},"assertionId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T12:56:07.244Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c","sourceUrl":"https://arxiv.org/abs/2611.06631","contentSha256":"ed565a5f5a0e942ee2160dbb8a24f974465b4127365bb5b2d2e05daee6ae80a0","textHash":"069473adfe0c403bf483b2a206ee3666c8a15f4b37e6c15f6195137826b46dfb","observedAt":"2026-11-09T12:56:07.244Z"}},{"actor":{"name":"Lee, Eujin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Lee, Eujin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06631","locator":"head/meta[name=author|citation_author]"},"assertionId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T12:56:07.244Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c","sourceUrl":"https://arxiv.org/abs/2611.06631","contentSha256":"ed565a5f5a0e942ee2160dbb8a24f974465b4127365bb5b2d2e05daee6ae80a0","textHash":"069473adfe0c403bf483b2a206ee3666c8a15f4b37e6c15f6195137826b46dfb","observedAt":"2026-11-09T12:56:07.244Z"}},{"actor":{"name":"Kim, Sung Eun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Kim, Sung Eun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06631","locator":"head/meta[name=author|citation_author]"},"assertionId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T12:56:07.244Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c","sourceUrl":"https://arxiv.org/abs/2611.06631","contentSha256":"ed565a5f5a0e942ee2160dbb8a24f974465b4127365bb5b2d2e05daee6ae80a0","textHash":"069473adfe0c403bf483b2a206ee3666c8a15f4b37e6c15f6195137826b46dfb","observedAt":"2026-11-09T12:56:07.244Z"}},{"actor":{"name":"Willauer, Alexandra N.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Willauer, Alexandra N.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06631","locator":"head/meta[name=author|citation_author]"},"assertionId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T12:56:07.244Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c","sourceUrl":"https://arxiv.org/abs/2611.06631","contentSha256":"ed565a5f5a0e942ee2160dbb8a24f974465b4127365bb5b2d2e05daee6ae80a0","textHash":"069473adfe0c403bf483b2a206ee3666c8a15f4b37e6c15f6195137826b46dfb","observedAt":"2026-11-09T12:56:07.244Z"}},{"actor":{"name":"Zhou, Christine Y.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhou, Christine Y.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06631","locator":"head/meta[name=author|citation_author]"},"assertionId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T12:56:07.244Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c","sourceUrl":"https://arxiv.org/abs/2611.06631","contentSha256":"ed565a5f5a0e942ee2160dbb8a24f974465b4127365bb5b2d2e05daee6ae80a0","textHash":"069473adfe0c403bf483b2a206ee3666c8a15f4b37e6c15f6195137826b46dfb","observedAt":"2026-11-09T12:56:07.244Z"}},{"actor":{"name":"Yang, Zhizhou","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yang, Zhizhou","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06631","locator":"head/meta[name=author|citation_author]"},"assertionId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T12:56:07.244Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c","sourceUrl":"https://arxiv.org/abs/2611.06631","contentSha256":"ed565a5f5a0e942ee2160dbb8a24f974465b4127365bb5b2d2e05daee6ae80a0","textHash":"069473adfe0c403bf483b2a206ee3666c8a15f4b37e6c15f6195137826b46dfb","observedAt":"2026-11-09T12:56:07.244Z"}},{"actor":{"name":"Rajpurkar, Pranav","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Rajpurkar, Pranav","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06631","locator":"head/meta[name=author|citation_author]"},"assertionId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T12:56:07.244Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c","sourceUrl":"https://arxiv.org/abs/2611.06631","contentSha256":"ed565a5f5a0e942ee2160dbb8a24f974465b4127365bb5b2d2e05daee6ae80a0","textHash":"069473adfe0c403bf483b2a206ee3666c8a15f4b37e6c15f6195137826b46dfb","observedAt":"2026-11-09T12:56:07.244Z"}}],"participationCoverage":{"total":7,"returned":7,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.06631","snapshotId":"e4461870967b7f4e5d897c1443f9fd684dc6b42934e37c1a356f0bb3e68b2a7c","contentSha256":"ed565a5f5a0e942ee2160dbb8a24f974465b4127365bb5b2d2e05daee6ae80a0","textHash":"069473adfe0c403bf483b2a206ee3666c8a15f4b37e6c15f6195137826b46dfb","observedAt":"2026-11-09T12:56:07.244Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/1d4f1ea149682ee84258c32e6044675e3a776fdf7b2a1c1e8d0c50f545f9beb6/source"}}}],"audits":[{"publicationId":"30b1e63e-409b-4667-87a3-c1a3be59763c","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading consistently attributes the benchmark, rates and interpretations to the paper's authors. The arXiv listing names Banerjee, Rajpurkar and colleagues. The title's 'arXiv cs.AI' matches the feed publisher; the paper is primarily cs.CV and cross-listed in cs.AI, which is acceptable.","excerpt":"Authors:Oishi Banerjee, Eujin Lee, Sung Eun Kim, Alexandra N. Willauer, Christine Y. Zhou, Zhizhou Yang, Pranav Rajpurkar"},{"dimension":"summary","status":"pass","explanation":"The summary explains the setup in plain language: real chest X-ray images paired with synthetic photos that either agree or conflict. It gives the headline rates and the acne-as-chickenpox example, and it keeps the scope limited to three unnamed models. Terms such as frontier models and hallucinated are kept exact for tooltips. Optional repair: restoring the authors' own framing of a 'concerning error pattern' would preserve their force, but the omission is not a fidelity failure. The question follows from the abstract's lack of per-model results.","excerpt":"Trick-or-Treat both demonstrates concerning error pattern in frontier models' medical reasoning"},{"dimension":"date","status":"pass","explanation":"originalPublished is 2026-11-04. This matches the arXiv submission date for v1. The feed date, 2026-11-05, was correctly not used.","excerpt":"[Submitted on 4 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"All claims match the abstract verbatim in substance. These include the benchmark design, the matched and mismatched pairs, the 45-57.5% versus 25-37.5% rates, the tendency to hallucinate findings that fit the radiograph, and the proof-of-concept framing. The limitations accurately note the unnamed models and the gap to clinical practice without inventing stakes.","excerpt":"models hallucinated clinically significant photograph findings in 45-57.5% of mismatched pairs, up from 25-37.5% for the same photographs shown alone."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the arXiv abstract page. It is the original source, and no better source media was found.","excerpt":null}],"createdAt":"2026-11-09T12:24:52.014Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}