Ill, Hannes, Attrach, Rafi Al, Fani, Rajna, Han, Ahram, Yamamoto, Ryohei, Moerig, Marlene Louisa, Yoon, Dukyong, Celi, Leo Anthony · Oct 27, 2026
Ill and colleagues test how AI coding agents derive clinical variables from EHR data
Markdown · JSON · Original source
Hannes Ill and coauthors introduce M4Bench, a for testing whether AI coding agents can reproduce clinical research variables derived from electronic health records. These variables depend on choices such as which item codes and windows to use, how to handle missing data, and what each output row represents. In a Docker-isolated Codex campaign, clinician-reviewed skills raised reward compared with no skills. The authors report that clearer, operational specifications accounted for most of the gain in a control subset; adding skills to those specifications contributed little. A descriptive Code replication showed a similar direction of change. The gains were partial agreement with reference results, not reliable task completion: few runs passed every strict check, and skills barely changed which rows agents returned. The benchmark therefore highlights a distinction between helping an agent follow a sufficiently precise procedure and getting it to return the intended data. How these results carry over to clinical research workflows beyond the tested datasets and systems remains unknown.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 27, 2026 (2026-10-27).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-06T21:46:28.332Z. First observed: 2026-10-28T05:54:32.814Z. Retrieved: 2026-10-30T01:19:04.373Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The reported results come from tasks on MIMIC-IV and eICU and specified coding-agent campaigns; the abstract does not establish performance in other clinical research workflows. The authors characterize the Code replication as descriptive. The reported gains are partial agreement with reference results, not completed tasks.
Open question
Would clearer specifications improve strict-check completion in clinical research workflows beyond the datasets and systems tested?
Source-stated
Ill and coauthors introduce M4Bench, a benchmark of clinical derivation tasks on MIMIC-IV and eICU, to evaluate AI coding agents and the effect of clinician-reviewed skills.
We introduce M4Bench, 28 clinical derivation tasks across 15 task families on MIMIC-IV and eICU
Source-stated
In the Codex campaign, skills raised reward over the no-skill baseline, but the authors attribute most of the measured gain to specification completeness.
skills raise reward by 0.136 over the no-skill baseline, with 26 of 28 tasks improving. Specification completeness accounts for most of this gain
Source-stated
The authors report that skills barely changed which rows agents returned and that few runs passed every strict check.
only 25 of 280 with-skill and 13 of 280 no-skill Codex runs pass every strict check, and skills barely change which rows agents return (key F1 +0.003).
Source-stated
The abstract describes a directionally consistent Claude Code replication, while characterizing it as descriptive.
A descriptive Claude Code replication on all 28 tasks (Opus 4.7 and Sonnet 4.6, 560 runs) is directionally consistent
Source and contributor context
Ill, Hannes, Attrach, Rafi Al, Fani, Rajna, Han, Ahram, Yamamoto, Ryohei, Moerig, Marlene Louisa, Yoon, Dukyong, Celi, Leo Anthony · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: c3071564-7547-4846-8ca1-c4c0056c4cf9.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-06T21:46:28.332Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 7696880cb8e92f84076de13185aa87c5ebf806dd89f6282ee97bd5b6eca5a78a. Source text hash: d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac.
Revision and source-check history
Version 1 · 2026-11-06T21:46:28.332Z
Ill and colleagues test how AI coding agents derive clinical variables from EHR data
Hannes Ill and coauthors introduce M4Bench, a for testing whether AI coding agents can reproduce clinical research variables derived from electronic health records. These variables depend on choices such as which item codes and windows to use, how to handle missing data, and what each output row represents. In a Docker-isolated Codex campaign, clinician-reviewed skills raised reward compared with no skills. The authors report that clearer, operational specifications accounted for most of the gain in a control subset; adding skills to those specifications contributed little. A descriptive Code replication showed a similar direction of change. The gains were partial agreement with reference results, not reliable task completion: few runs passed every strict check, and skills barely changed which rows agents returned. The benchmark therefore highlights a distinction between helping an agent follow a sufficiently precise procedure and getting it to return the intended data. How these results carry over to clinical research workflows beyond the tested datasets and systems remains unknown.
Original source
cs.AI · Article
Source details
Original source date: Oct 27, 2026 (2026-10-27).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-06T21:46:28.332Z. First observed: 2026-10-28T05:54:32.814Z. Retrieved: 2026-10-30T01:19:04.373Z.
Limits
The reported results come from tasks on MIMIC-IV and eICU and specified coding-agent campaigns; the abstract does not establish performance in other clinical research workflows. The authors characterize the Code replication as descriptive. The reported gains are partial agreement with reference results, not completed tasks.
Open question
Would clearer specifications improve strict-check completion in clinical research workflows beyond the datasets and systems tested?
Source-stated
Ill and coauthors introduce M4Bench, a benchmark of clinical derivation tasks on MIMIC-IV and eICU, to evaluate AI coding agents and the effect of clinician-reviewed skills.
We introduce M4Bench, 28 clinical derivation tasks across 15 task families on MIMIC-IV and eICU
Source-stated
In the Codex campaign, skills raised reward over the no-skill baseline, but the authors attribute most of the measured gain to specification completeness.
skills raise reward by 0.136 over the no-skill baseline, with 26 of 28 tasks improving. Specification completeness accounts for most of this gain
Source-stated
The authors report that skills barely changed which rows agents returned and that few runs passed every strict check.
only 25 of 280 with-skill and 13 of 280 no-skill Codex runs pass every strict check, and skills barely change which rows agents return (key F1 +0.003).
Source-stated
The abstract describes a directionally consistent Claude Code replication, while characterizing it as descriptive.
A descriptive Claude Code replication on all 28 tasks (Opus 4.7 and Sonnet 4.6, 560 runs) is directionally consistent
Source and contributor context
Ill, Hannes, Attrach, Rafi Al, Fani, Rajna, Han, Ahram, Yamamoto, Ryohei, Moerig, Marlene Louisa, Yoon, Dukyong, Celi, Leo Anthony · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: c3071564-7547-4846-8ca1-c4c0056c4cf9.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-06T21:46:28.332Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 7696880cb8e92f84076de13185aa87c5ebf806dd89f6282ee97bd5b6eca5a78a. Source text hash: d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac.
uncertain · 2026-11-06T21:46:28.332Z · anthropic/claude-opus-5.5
Publication: c3071564-7547-4846-8ca1-c4c0056c4cf9
attribution — pass: The draft correctly credits Hannes Ill and coauthors and keeps the findings attributed to the authors, consistent with the arXiv author list and abstract.
Authors:Hannes Ill, Rafi Al Attrach, Rajna Fani, Ahram Han, Ryohei Yamamoto, Marlene Louisa Moerig, Dukyong Yoon, Leo Anthony Celi
summary — uncertain: The draft explains derived variables well, but leaves central terms unclear. "reward" is the core metric and is never defined. "skills" (clinician-reviewed markdown instructions given to the agent) is also unexplained, as are "Docker-isolated Codex campaign" and MIMIC-IV/eICU, so a non-specialist cannot tell what improved. "clearer, operational specifications" paraphrases the technical term "specification completeness"; restore that term. Smallest repair: briefly gloss reward as partial agreement with a reference result, skills as reviewed markdown instructions, and MIMIC-IV/eICU as clinical databases.
Skills are released as a versioned, clinician-reviewable markdown delivery format
date — pass: originalPublished 2026-10-27 matches the arXiv v1 submission date. The later feed date is correctly not used.
[Submitted on 27 Oct 2026]
support — uncertain: Most claims match the abstract. However, the sentence saying the benchmark "therefore highlights a distinction between helping an agent follow a sufficiently precise procedure and getting it to return the intended data" is an unlabeled inference. The abstract reports row-level change (key F1) only for skills, not for operational specs alone, so the distinction is not directly established. Smallest repair: label the sentence as Escape Meridian interpretation and make it conditional. The question is open and acceptable, although the abstract reports spec effects on reward, not on strict-check pass rates.
skills barely change which rows agents return (key F1 +0.003)
media — pass: The selected media is the verified arXiv abstract page for this paper, the direct original source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore Claude