Park, Christine, Chen, Valerie, Dettmers, Tim · Sep 24, 2026
arXiv cs.AI: Researchers introduce Synthetic Hospital, a synthetic benchmark for clinical AI
Markdown · JSON · Original source
The authors introduce Synthetic Hospital, an open, fully synthetic of longitudinal EHRs, designed to test whether AI models can track patient information across encounters. They argue that real EHR data is difficult to share openly and lacks verifiable ground truth: records show what clinicians documented, not necessarily everything that happened. Synthetic Hospital contains 1,268 patients and 5,602 encounters. Its diagnoses, findings and temporal relations are grounded in standard ontologies, with provenance linking them to public medical-education material. A simulated hospital record system provides access through interfaces that mirror real EHR infrastructure. In a blinded review, physicians identified the benchmark’s records as synthetic or real at near-chance rates. Yet the authors report that even the best model misses roughly half of clinically relevant findings when summarizing a chart. That model matched the mean of seven physicians on a matched subset when reconstructing longitudinal problem lists, but scored below the best physician. The authors present these results as evidence that the benchmark is a difficult test of clinical AI performance. The abstract does not establish how performance on this synthetic benchmark relates to performance in real clinical workflows.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Sep 24, 2026 (2026-09-24).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-14T05:05:39.816Z. First observed: 2026-09-25T07:40:35.728Z. Retrieved: 2026-10-30T21:31:38.023Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The abstract reports results but does not establish how performance on synthetic records predicts performance in real clinical workflows. The abstract does not describe the benchmark's coverage of patient populations or clinical settings; its generalizability beyond the reported evaluation is therefore unclear.
Open question
How well does performance on Synthetic Hospital predict model performance on real patient records and in clinical workflows?
Source-stated
The authors introduce Synthetic Hospital as an open, fully synthetic benchmark for longitudinal EHRs, intended to address barriers to sharing real records and establishing verifiable ground truth.
We introduce Synthetic Hospital, an open, fully synthetic, fact-grounded longitudinal EHR benchmark that resolves the open sharing and verifiable ground truth barriers.
Source-stated
The benchmark contains 1,268 longitudinal patients and 5,602 encounters, grounded in ontologies and linked to public medical-education material.
it comprises 1,268 longitudinal patients and 5,602 encounters, where every diagnosis, finding, and temporal relation is grounded in standard ontologies
Source-stated
Physicians distinguished the benchmark's records from real patient charts at near-chance rates in a blinded review.
In a blinded review, physicians distinguished its records from real patient charts at near-chance rates (53%).
Source-stated
The best model matched the mean of seven physicians on a matched subset for reconstructing a longitudinal problem list, but fell below the best physician.
the best model reconstructs a patient's longitudinal problem list with a severity-weighted F1 of 0.73, level with the mean of seven physicians on a matched subset but well below the best of them (0.89)
Source-stated
The authors report that models miss roughly half of clinically relevant findings when summarizing charts.
misses roughly half of clinically relevant findings when summarizing a chart
Source and contributor context
Park, Christine, Chen, Valerie, Dettmers, Tim · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 07cc7b09-ec20-45e0-8288-c732ce478df9.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-14T05:05:39.816Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 0a2a28ce773e4370d3e29c27f3f6e47974c28d830f5a0d91556b0861a7d549dd. Source text hash: e351d1201e997647059a5b3e42ef66ec9a415484c9ae395e094c3d9ee25da52e.
Revision and source-check history
Version 1 · 2026-11-14T05:05:39.816Z
arXiv cs.AI: Researchers introduce Synthetic Hospital, a synthetic benchmark for clinical AI
The authors introduce Synthetic Hospital, an open, fully synthetic of longitudinal EHRs, designed to test whether AI models can track patient information across encounters. They argue that real EHR data is difficult to share openly and lacks verifiable ground truth: records show what clinicians documented, not necessarily everything that happened. Synthetic Hospital contains 1,268 patients and 5,602 encounters. Its diagnoses, findings and temporal relations are grounded in standard ontologies, with provenance linking them to public medical-education material. A simulated hospital record system provides access through interfaces that mirror real EHR infrastructure. In a blinded review, physicians identified the benchmark’s records as synthetic or real at near-chance rates. Yet the authors report that even the best model misses roughly half of clinically relevant findings when summarizing a chart. That model matched the mean of seven physicians on a matched subset when reconstructing longitudinal problem lists, but scored below the best physician. The authors present these results as evidence that the benchmark is a difficult test of clinical AI performance. The abstract does not establish how performance on this synthetic benchmark relates to performance in real clinical workflows.
Original source
cs.AI · Article
Source details
Original source date: Sep 24, 2026 (2026-09-24).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-14T05:05:39.816Z. First observed: 2026-09-25T07:40:35.728Z. Retrieved: 2026-10-30T21:31:38.023Z.
Limits
The abstract reports results but does not establish how performance on synthetic records predicts performance in real clinical workflows. The abstract does not describe the benchmark's coverage of patient populations or clinical settings; its generalizability beyond the reported evaluation is therefore unclear.
Open question
How well does performance on Synthetic Hospital predict model performance on real patient records and in clinical workflows?
Source-stated
The authors introduce Synthetic Hospital as an open, fully synthetic benchmark for longitudinal EHRs, intended to address barriers to sharing real records and establishing verifiable ground truth.
We introduce Synthetic Hospital, an open, fully synthetic, fact-grounded longitudinal EHR benchmark that resolves the open sharing and verifiable ground truth barriers.
Source-stated
The benchmark contains 1,268 longitudinal patients and 5,602 encounters, grounded in ontologies and linked to public medical-education material.
it comprises 1,268 longitudinal patients and 5,602 encounters, where every diagnosis, finding, and temporal relation is grounded in standard ontologies
Source-stated
Physicians distinguished the benchmark's records from real patient charts at near-chance rates in a blinded review.
In a blinded review, physicians distinguished its records from real patient charts at near-chance rates (53%).
Source-stated
The best model matched the mean of seven physicians on a matched subset for reconstructing a longitudinal problem list, but fell below the best physician.
the best model reconstructs a patient's longitudinal problem list with a severity-weighted F1 of 0.73, level with the mean of seven physicians on a matched subset but well below the best of them (0.89)
Source-stated
The authors report that models miss roughly half of clinically relevant findings when summarizing charts.
misses roughly half of clinically relevant findings when summarizing a chart
Source and contributor context
Park, Christine, Chen, Valerie, Dettmers, Tim · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 07cc7b09-ec20-45e0-8288-c732ce478df9.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-14T05:05:39.816Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 0a2a28ce773e4370d3e29c27f3f6e47974c28d830f5a0d91556b0861a7d549dd. Source text hash: e351d1201e997647059a5b3e42ef66ec9a415484c9ae395e094c3d9ee25da52e.
uncertain · 2026-11-14T05:05:39.816Z · anthropic/claude-opus-5.5
Publication: 07cc7b09-ec20-45e0-8288-c732ce478df9
attribution — pass: The draft attributes the benchmark, its design rationale and its results to the paper's authors (Park, Chen, Dettmers) on arXiv cs.AI. The 'difficult test' conclusion is correctly framed as the authors' own presentation, not as an established fact.
Authors:Christine Park, Valerie Chen, Tim Dettmers
summary — uncertain: Two gaps affect a lay reader. First, 'EHR' is never expanded to 'electronic health record', though it is the central term; add the expansion or a tooltip. Second, the draft omits the authors' framing of why this matters: frontier models are rarely used clinically because realistic benchmarks are scarce. One clause can restore it. It also softens 'well below the best' to 'below' and drops 'none approaches ceiling' and the 0.73 vs 0.89 gap. Smallest repair: restore 'well below' and note the result covers 10 models.
Frontier language models are rarely used in clinical workflows because the realistic, longitudinal benchmarks needed to develop them are scarce.
date — pass: originalPublished 2026-09-24 matches the v1 submission date, not the later v2 revision (27 Oct) or the feed date (29 Oct).
[Submitted on 24 Sep 2026 (v1), last revised 27 Oct 2026 (this version, v2)]
support — pass: Counts, grounding, blinded review and model-versus-physician comparisons match the abstract. Claim 5 says 'models miss roughly half', but the abstract attributes this to the best model. This is a minor broadening; the summary body states it correctly. The limitations and the question are appropriately conditional and do not overstate real-world reliability.
misses roughly half of clinically relevant findings when summarizing a chart
media — pass: The selected media is the verified arXiv abstract page for this paper. It is the direct original source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference