Lee, Simon A., Sha, Jiahang, Jeong, Moonseong, Pan, Yunbei, Chiang, Jeffrey N. · Oct 26, 2026
MedJev: Evaluating Zero-Shot System-One Decision Making in Medicine
Markdown · JSON · Original source
The authors of MedJev report evaluating an open System One model that runs locally on a laptop and answers structured questions in a single forward pass, without generating text. Across ten medical , they report competitive results on medical knowledge and diagnosis tasks, but a more mixed picture for patient-level prediction: MedJev exceeded XGBoost on In Hospital Mortality AUROC, while underperforming supervised models on ED Triage Admission. The authors also report substantial miscalibration on In Hospital Mortality, meaning its confidence estimates did not align well with observed outcomes. Their lightweight recalibration method uses labeled examples from a local patient population to reduce that mismatch. The results make MedJev a research baseline for examining whether locally run models can support medical prediction at low computational cost; the abstract does not establish clinical effectiveness. How well does the reported performance and recalibration hold up across hospitals and in clinical use?
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 26, 2026 (2026-10-26).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T20:33:01.515Z. First observed: 2026-10-28T07:28:45.934Z. Retrieved: 2026-10-30T21:03:42.742Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The supplied evidence is the abstract, not the full paper, so it does not provide details of methods or results beyond those stated there. The abstract reports benchmark evaluations, not clinical effectiveness in use. The abstract does not establish how performance or recalibration generalizes across hospitals.
Open question
How well does the reported performance and recalibration hold up across hospitals and in clinical use?
Source-stated
The authors describe System One decision models as answering structured questions in a single forward pass without generating text, and say MedJev runs locally on a laptop.
answer structured questions in a single forward pass without generating text
Source-stated
The authors evaluated MedJev across ten medical benchmarks spanning medical knowledge, diagnosis, and patient-level prediction.
across ten medical benchmarks spanning medical knowledge, diagnosis, and patient level prediction
Source-stated
The authors report 72.5 percent accuracy on MedQA, 75.4 percent on USMLE, and 80.1 percent on NEJM.
accuracies of 72.5 percent on MedQA, 75.4 percent on USMLE, and 80.1 percent on NEJM
Source-stated
For In Hospital Mortality, the authors report an AUROC of 0.926 for MedJev, compared with 0.902 for XGBoost.
AUROC of 0.926 for In Hospital Mortality, exceeding XGBoost (0.902)
Source-stated
For ED Triage Admission, the authors report that MedJev underperformed supervised models, with results of 0.745 versus 0.809.
underperforms supervised models on operational outcomes such as ED Triage Admission (0.745 vs. 0.809)
Escape Meridian interpretation
The abstract reports benchmark results and a recalibration method; it does not report evidence of clinical effectiveness in use.
We evaluate MedJev, an open System One model that runs locally on a laptop
Source and contributor context
Lee, Simon A., Sha, Jiahang, Jeong, Moonseong, Pan, Yunbei, Chiang, Jeffrey N. · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Zero-shot
Doing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 71180004-0eab-4ac7-8be8-ee8a90ab2c7b.
Prepared by server-excerpt-repair/stream-excerpt-repair-v1 (original draft: gpt-6-luna); grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T20:33:01.515Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 612094e9188f4cc561909c0299cc786d30a30baa7fa05eee5fb4cd6ee3d8d81c. Source text hash: e591f52ef3e0deb9a46ab0c360d544bc65d7f0ae6879b996b238276cf65aaced.
Revision and source-check history
Version 1 · 2026-11-10T20:33:01.515Z
MedJev: Evaluating Zero-Shot System-One Decision Making in Medicine
The authors of MedJev report evaluating an open System One model that runs locally on a laptop and answers structured questions in a single forward pass, without generating text. Across ten medical , they report competitive results on medical knowledge and diagnosis tasks, but a more mixed picture for patient-level prediction: MedJev exceeded XGBoost on In Hospital Mortality AUROC, while underperforming supervised models on ED Triage Admission. The authors also report substantial miscalibration on In Hospital Mortality, meaning its confidence estimates did not align well with observed outcomes. Their lightweight recalibration method uses labeled examples from a local patient population to reduce that mismatch. The results make MedJev a research baseline for examining whether locally run models can support medical prediction at low computational cost; the abstract does not establish clinical effectiveness. How well does the reported performance and recalibration hold up across hospitals and in clinical use?
Original source
cs.AI · Article
Source details
Original source date: Oct 26, 2026 (2026-10-26).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T20:33:01.515Z. First observed: 2026-10-28T07:28:45.934Z. Retrieved: 2026-10-30T21:03:42.742Z.
Limits
The supplied evidence is the abstract, not the full paper, so it does not provide details of methods or results beyond those stated there. The abstract reports benchmark evaluations, not clinical effectiveness in use. The abstract does not establish how performance or recalibration generalizes across hospitals.
Open question
How well does the reported performance and recalibration hold up across hospitals and in clinical use?
Source-stated
The authors describe System One decision models as answering structured questions in a single forward pass without generating text, and say MedJev runs locally on a laptop.
answer structured questions in a single forward pass without generating text
Source-stated
The authors evaluated MedJev across ten medical benchmarks spanning medical knowledge, diagnosis, and patient-level prediction.
across ten medical benchmarks spanning medical knowledge, diagnosis, and patient level prediction
Source-stated
The authors report 72.5 percent accuracy on MedQA, 75.4 percent on USMLE, and 80.1 percent on NEJM.
accuracies of 72.5 percent on MedQA, 75.4 percent on USMLE, and 80.1 percent on NEJM
Source-stated
For In Hospital Mortality, the authors report an AUROC of 0.926 for MedJev, compared with 0.902 for XGBoost.
AUROC of 0.926 for In Hospital Mortality, exceeding XGBoost (0.902)
Source-stated
For ED Triage Admission, the authors report that MedJev underperformed supervised models, with results of 0.745 versus 0.809.
underperforms supervised models on operational outcomes such as ED Triage Admission (0.745 vs. 0.809)
Escape Meridian interpretation
The abstract reports benchmark results and a recalibration method; it does not report evidence of clinical effectiveness in use.
We evaluate MedJev, an open System One model that runs locally on a laptop
Source and contributor context
Lee, Simon A., Sha, Jiahang, Jeong, Moonseong, Pan, Yunbei, Chiang, Jeffrey N. · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Zero-shot
Doing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 71180004-0eab-4ac7-8be8-ee8a90ab2c7b.
Prepared by server-excerpt-repair/stream-excerpt-repair-v1 (original draft: gpt-6-luna); grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T20:33:01.515Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 612094e9188f4cc561909c0299cc786d30a30baa7fa05eee5fb4cd6ee3d8d81c. Source text hash: e591f52ef3e0deb9a46ab0c360d544bc65d7f0ae6879b996b238276cf65aaced.
uncertain · 2026-11-10T20:33:01.515Z · anthropic/claude-opus-5.5
Publication: 71180004-0eab-4ac7-8be8-ee8a90ab2c7b
attribution — pass: The reading attributes the evaluation, results and recalibration method to the MedJev paper's authors. The authors are listed on the arXiv page, and the reading does not present their claims as established fact.
Authors:Simon A. Lee, Jiahang Sha, Moonseong Jeong, Yunbei Pan, Jeffrey N. Chiang
summary — uncertain: The reading is broadly faithful but has three fidelity gaps. First, it drops the authors' central benchmark claim that MedJev approaches published GPT-4 performance with well-calibrated knowledge-task confidence. Second, it softens their conclusion that open System One models are a promising zero-shot baseline into an unattributed EM framing. Third, AUROC, XGBoost and zero-shot are central but unexplained for lay readers. Smallest repair: restore the attributed GPT-4 comparison and the authors' promising-baseline conclusion, and add term tooltips.
approaching published GPT 4 performance while maintaining well calibrated confidence estimates (ECE 0.028 to 0.063)
date — pass: The originalPublished date, 2026-10-26, matches the v1 submission date on arXiv. The feed date is later and is correctly not used.
[Submitted on 26 Oct 2026]
support — pass: The figures, comparisons, miscalibration finding and recalibration description all match the abstract. The labeled interpretation, that the abstract shows benchmark results rather than clinical effectiveness, is a fair limitation. The question about generalization across hospitals follows from the local-recalibration design. The word competitive for diagnosis tasks is loosely grounded but not contradicted.
reducing ECE to 0.007
media — pass: The selected media item is the verified direct arXiv abstract page for this paper, which is the most relevant way to access the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference