# MedJev: Evaluating Zero-Shot System-One Decision Making in Medicine [Canonical reading](https://escapemeridian.com/readings/medjev-evaluating-zero-shot-system-one-decision-making-in-c777657e) The authors of MedJev report evaluating an open System One model that runs locally on a laptop and answers structured questions in a single forward pass, without generating text. Across ten medical benchmarks, they report competitive results on medical knowledge and diagnosis tasks, but a more mixed picture for patient-level prediction: MedJev exceeded XGBoost on In Hospital Mortality AUROC, while underperforming supervised models on ED Triage Admission. The authors also report substantial miscalibration on In Hospital Mortality, meaning its confidence estimates did not align well with observed outcomes. Their lightweight recalibration method uses labeled examples from a local patient population to reduce that mismatch. The results make MedJev a research baseline for examining whether locally run models can support medical prediction at low computational cost; the abstract does not establish clinical effectiveness. How well does the reported performance and recalibration hold up across hospitals and in clinical use? Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Oct 26, 2026 (2026-10-26). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-10T20:33:01.515Z. First observed: 2026-10-28T07:28:45.934Z. Retrieved: 2026-10-30T21:03:42.742Z. - [MedJev: Evaluating Zero-Shot System-One Decision Making in Medicine](https://arxiv.org/abs/2610.36006) ## Limits The supplied evidence is the arXiv abstract, not the full paper, so it does not provide details of benchmark methods or results beyond those stated there. The abstract reports benchmark evaluations, not clinical effectiveness in use. The abstract does not establish how performance or recalibration generalizes across hospitals. ## Open question How well does the reported performance and recalibration hold up across hospitals and in clinical use? ## Source-stated The authors describe System One decision models as answering structured questions in a single forward pass without generating text, and say MedJev runs locally on a laptop. > answer structured questions in a single forward pass without generating text - [Supporting source](https://arxiv.org/abs/2610.36006) ## Source-stated The authors evaluated MedJev across ten medical benchmarks spanning medical knowledge, diagnosis, and patient-level prediction. > across ten medical benchmarks spanning medical knowledge, diagnosis, and patient level prediction - [Supporting source](https://arxiv.org/abs/2610.36006) ## Source-stated The authors report 72.5 percent accuracy on MedQA, 75.4 percent on USMLE, and 80.1 percent on NEJM. > accuracies of 72.5 percent on MedQA, 75.4 percent on USMLE, and 80.1 percent on NEJM - [Supporting source](https://arxiv.org/abs/2610.36006) ## Source-stated For In Hospital Mortality, the authors report an AUROC of 0.926 for MedJev, compared with 0.902 for XGBoost. > AUROC of 0.926 for In Hospital Mortality, exceeding XGBoost (0.902) - [Supporting source](https://arxiv.org/abs/2610.36006) ## Source-stated For ED Triage Admission, the authors report that MedJev underperformed supervised models, with results of 0.745 versus 0.809. > underperforms supervised models on operational outcomes such as ED Triage Admission (0.745 vs. 0.809) - [Supporting source](https://arxiv.org/abs/2610.36006) ## Escape Meridian interpretation The abstract reports benchmark results and a recalibration method; it does not report evidence of clinical effectiveness in use. > We evaluate MedJev, an open System One model that runs locally on a laptop - [Supporting source](https://arxiv.org/abs/2610.36006) ## Source and contributor context Lee, Simon A., Sha, Jiahang, Jeong, Moonseong, Pan, Yunbei, Chiang, Jeffrey N. · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Zero-shot Doing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot). - [Definition reference](https://developers.google.com/machine-learning/glossary#zero-shot-prompting) ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 71180004-0eab-4ac7-8be8-ee8a90ab2c7b. Prepared by server-excerpt-repair/stream-excerpt-repair-v1 (original draft: gpt-6-luna); grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T20:33:01.515Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 612094e9188f4cc561909c0299cc786d30a30baa7fa05eee5fb4cd6ee3d8d81c. Source text hash: e591f52ef3e0deb9a46ab0c360d544bc65d7f0ae6879b996b238276cf65aaced. ## Revision and source-check history ### Version 1 — 2026-11-10T20:33:01.515Z The authors of MedJev report evaluating an open System One model that runs locally on a laptop and answers structured questions in a single forward pass, without generating text. Across ten medical benchmarks, they report competitive results on medical knowledge and diagnosis tasks, but a more mixed picture for patient-level prediction: MedJev exceeded XGBoost on In Hospital Mortality AUROC, while underperforming supervised models on ED Triage Admission. The authors also report substantial miscalibration on In Hospital Mortality, meaning its confidence estimates did not align well with observed outcomes. Their lightweight recalibration method uses labeled examples from a local patient population to reduce that mismatch. The results make MedJev a research baseline for examining whether locally run models can support medical prediction at low computational cost; the abstract does not establish clinical effectiveness. How well does the reported performance and recalibration hold up across hospitals and in clinical use? ## Original source arXiv cs.AI · Article Original source date: Oct 26, 2026 (2026-10-26). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-10T20:33:01.515Z. First observed: 2026-10-28T07:28:45.934Z. Retrieved: 2026-10-30T21:03:42.742Z. - [MedJev: Evaluating Zero-Shot System-One Decision Making in Medicine](https://arxiv.org/abs/2610.36006) ## Limits The supplied evidence is the arXiv abstract, not the full paper, so it does not provide details of benchmark methods or results beyond those stated there. The abstract reports benchmark evaluations, not clinical effectiveness in use. The abstract does not establish how performance or recalibration generalizes across hospitals. ## Open question How well does the reported performance and recalibration hold up across hospitals and in clinical use? ## Source-stated The authors describe System One decision models as answering structured questions in a single forward pass without generating text, and say MedJev runs locally on a laptop. > answer structured questions in a single forward pass without generating text - [Supporting source](https://arxiv.org/abs/2610.36006) ## Source-stated The authors evaluated MedJev across ten medical benchmarks spanning medical knowledge, diagnosis, and patient-level prediction. > across ten medical benchmarks spanning medical knowledge, diagnosis, and patient level prediction - [Supporting source](https://arxiv.org/abs/2610.36006) ## Source-stated The authors report 72.5 percent accuracy on MedQA, 75.4 percent on USMLE, and 80.1 percent on NEJM. > accuracies of 72.5 percent on MedQA, 75.4 percent on USMLE, and 80.1 percent on NEJM - [Supporting source](https://arxiv.org/abs/2610.36006) ## Source-stated For In Hospital Mortality, the authors report an AUROC of 0.926 for MedJev, compared with 0.902 for XGBoost. > AUROC of 0.926 for In Hospital Mortality, exceeding XGBoost (0.902) - [Supporting source](https://arxiv.org/abs/2610.36006) ## Source-stated For ED Triage Admission, the authors report that MedJev underperformed supervised models, with results of 0.745 versus 0.809. > underperforms supervised models on operational outcomes such as ED Triage Admission (0.745 vs. 0.809) - [Supporting source](https://arxiv.org/abs/2610.36006) ## Escape Meridian interpretation The abstract reports benchmark results and a recalibration method; it does not report evidence of clinical effectiveness in use. > We evaluate MedJev, an open System One model that runs locally on a laptop - [Supporting source](https://arxiv.org/abs/2610.36006) ## Source and contributor context Lee, Simon A., Sha, Jiahang, Jeong, Moonseong, Pan, Yunbei, Chiang, Jeffrey N. · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Zero-shot Doing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot). - [Definition reference](https://developers.google.com/machine-learning/glossary#zero-shot-prompting) ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 71180004-0eab-4ac7-8be8-ee8a90ab2c7b. Prepared by server-excerpt-repair/stream-excerpt-repair-v1 (original draft: gpt-6-luna); grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T20:33:01.515Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 612094e9188f4cc561909c0299cc786d30a30baa7fa05eee5fb4cd6ee3d8d81c. Source text hash: e591f52ef3e0deb9a46ab0c360d544bc65d7f0ae6879b996b238276cf65aaced. ### uncertain — 2026-11-10T20:33:01.515Z — anthropic/claude-opus-5.5 Publication: 71180004-0eab-4ac7-8be8-ee8a90ab2c7b attribution — pass: The reading attributes the evaluation, results and recalibration method to the MedJev paper's authors. The authors are listed on the arXiv page, and the reading does not present their claims as established fact. > Authors:Simon A. Lee, Jiahang Sha, Moonseong Jeong, Yunbei Pan, Jeffrey N. Chiang summary — uncertain: The reading is broadly faithful but has three fidelity gaps. First, it drops the authors' central benchmark claim that MedJev approaches published GPT-4 performance with well-calibrated knowledge-task confidence. Second, it softens their conclusion that open System One models are a promising zero-shot baseline into an unattributed EM framing. Third, AUROC, XGBoost and zero-shot are central but unexplained for lay readers. Smallest repair: restore the attributed GPT-4 comparison and the authors' promising-baseline conclusion, and add term tooltips. > approaching published GPT 4 performance while maintaining well calibrated confidence estimates (ECE 0.028 to 0.063) date — pass: The originalPublished date, 2026-10-26, matches the v1 submission date on arXiv. The feed date is later and is correctly not used. > \[Submitted on 26 Oct 2026\] support — pass: The figures, comparisons, miscalibration finding and recalibration description all match the abstract. The labeled interpretation, that the abstract shows benchmark results rather than clinical effectiveness, is a fair limitation. The question about generalization across hospitals follows from the local-recalibration design. The word competitive for diagnosis tasks is loosely grounded but not contradicted. > reducing ECE to 0.007 media — pass: The selected media item is the verified direct arXiv abstract page for this paper, which is the most relevant way to access the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/c777657e62a0012ef756da6d7d54226f4ec8fddde3c53fbb09ed15cceae09d6a/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/medjev-evaluating-zero-shot-system-one-decision-making-in-c777657e.json)