{"id":"638c56be70b3c9795cff635aced0b4d62a06227910c80f26d810f420f2729d0c","title":"Ill and colleagues test how AI coding agents derive clinical variables from EHR data","argument":"Hannes Ill and coauthors introduce M4Bench, a benchmark for testing whether AI coding agents can reproduce clinical research variables derived from electronic health records. These variables depend on choices such as which item codes and time windows to use, how to handle missing data, and what each output row represents.\n\nIn a Docker-isolated Codex campaign, clinician-reviewed skills raised reward compared with no skills. The authors report that clearer, operational specifications accounted for most of the gain in a control subset; adding skills to those specifications contributed little. A descriptive Claude Code replication showed a similar direction of change.\n\nThe gains were partial agreement with reference results, not reliable task completion: few runs passed every strict check, and skills barely changed which rows agents returned. The benchmark therefore highlights a distinction between helping an agent follow a sufficiently precise procedure and getting it to return the intended data. How these results carry over to clinical research workflows beyond the tested datasets and systems remains unknown.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported results come from benchmark tasks on MIMIC-IV and eICU and specified coding-agent campaigns; the abstract does not establish performance in other clinical research workflows. The authors characterize the Claude Code replication as descriptive. The reported gains are partial agreement with reference results, not completed tasks.","speaker":{"name":"Ill, Hannes, Attrach, Rafi Al, Fani, Rajna, Han, Ahram, Yamamoto, Ryohei, Moerig, Marlene Louisa, Yoon, Dukyong, Celi, Leo Anthony","roleAtSource":"Article author"},"source":{"title":"M4Bench: Evaluating Procedural Specification for Clinical EHR Derivation Agents","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.37459","publishedAt":"2026-10-27","dateBasis":"Date basis not provided."},"evidence":[{"text":"Ill and coauthors introduce M4Bench, a benchmark of clinical derivation tasks on MIMIC-IV and eICU, to evaluate AI coding agents and the effect of clinician-reviewed skills.","excerpt":"We introduce M4Bench, 28 clinical derivation tasks across 15 task families on MIMIC-IV and eICU","url":"https://arxiv.org/abs/2610.37459","relationship":"Source-stated"},{"text":"In the Codex campaign, skills raised reward over the no-skill baseline, but the authors attribute most of the measured gain to specification completeness.","excerpt":"skills raise reward by 0.136 over the no-skill baseline, with 26 of 28 tasks improving. Specification completeness accounts for most of this gain","url":"https://arxiv.org/abs/2610.37459","relationship":"Source-stated"},{"text":"The authors report that skills barely changed which rows agents returned and that few runs passed every strict check.","excerpt":"only 25 of 280 with-skill and 13 of 280 no-skill Codex runs pass every strict check, and skills barely change which rows agents return (key F1 +0.003).","url":"https://arxiv.org/abs/2610.37459","relationship":"Source-stated"},{"text":"The abstract describes a directionally consistent Claude Code replication, while characterizing it as descriptive.","excerpt":"A descriptive Claude Code replication on all 28 tasks (Opus 4.7 and Sonnet 4.6, 560 runs) is directionally consistent","url":"https://arxiv.org/abs/2610.37459","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.37459","checkedAt":"2026-11-06T22:50:09.637Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Would clearer specifications improve strict-check completion in clinical research workflows beyond the datasets and systems tested?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Ill and colleagues test how AI coding agents derive clinical variables from EHR data","summary":"Hannes Ill and coauthors introduce M4Bench, a benchmark for testing whether AI coding agents can reproduce clinical research variables derived from electronic health records. These variables depend on choices such as which item codes and time windows to use, how to handle missing data, and what each output row represents.\n\nIn a Docker-isolated Codex campaign, clinician-reviewed skills raised reward compared with no skills. The authors report that clearer, operational specifications accounted for most of the gain in a control subset; adding skills to those specifications contributed little. A descriptive Claude Code replication showed a similar direction of change.\n\nThe gains were partial agreement with reference results, not reliable task completion: few runs passed every strict check, and skills barely changed which rows agents returned. The benchmark therefore highlights a distinction between helping an agent follow a sufficiently precise procedure and getting it to return the intended data. How these results carry over to clinical research workflows beyond the tested datasets and systems remains unknown.","question":"Would clearer specifications improve strict-check completion in clinical research workflows beyond the datasets and systems tested?","claims":[{"text":"Ill and coauthors introduce M4Bench, a benchmark of clinical derivation tasks on MIMIC-IV and eICU, to evaluate AI coding agents and the effect of clinician-reviewed skills.","excerpt":"We introduce M4Bench, 28 clinical derivation tasks across 15 task families on MIMIC-IV and eICU","support":"source-stated"},{"text":"In the Codex campaign, skills raised reward over the no-skill baseline, but the authors attribute most of the measured gain to specification completeness.","excerpt":"skills raise reward by 0.136 over the no-skill baseline, with 26 of 28 tasks improving. Specification completeness accounts for most of this gain","support":"source-stated"},{"text":"The authors report that skills barely changed which rows agents returned and that few runs passed every strict check.","excerpt":"only 25 of 280 with-skill and 13 of 280 no-skill Codex runs pass every strict check, and skills barely change which rows agents return (key F1 +0.003).","support":"source-stated"},{"text":"The abstract describes a directionally consistent Claude Code replication, while characterizing it as descriptive.","excerpt":"A descriptive Claude Code replication on all 28 tasks (Opus 4.7 and Sonnet 4.6, 560 runs) is directionally consistent","support":"source-stated"}],"limitations":["The reported results come from benchmark tasks on MIMIC-IV and eICU and specified coding-agent campaigns; the abstract does not establish performance in other clinical research workflows.","The authors characterize the Claude Code replication as descriptive.","The reported gains are partial agreement with reference results, not completed tasks."],"mediaId":"fec5f639cc2380fb76770a155f83d6a1753a8b1ca8c4049ccdfecdd210dc6961","correctionReason":null,"id":"638c56be70b3c9795cff635aced0b4d62a06227910c80f26d810f420f2729d0c","publicationId":"c3071564-7547-4846-8ca1-c4c0056c4cf9","version":1,"sourceUrl":"https://arxiv.org/abs/2610.37459","sourceTitle":"M4Bench: Evaluating Procedural Specification for Clinical EHR Derivation Agents","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-27","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-28T05:54:32.814Z","createdAt":"2026-11-06T21:46:28.332Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"fec5f639cc2380fb76770a155f83d6a1753a8b1ca8c4049ccdfecdd210dc6961","kind":"article","url":"https://arxiv.org/abs/2610.37459","sourceUrl":"https://arxiv.org/abs/2610.37459","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-30T01:19:04.866Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.37459","checkedAt":"2026-11-06T22:50:09.637Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"fec5f639cc2380fb76770a155f83d6a1753a8b1ca8c4049ccdfecdd210dc6961","confidence":0.83,"provider":"typesafe","model":"jev-1.13.0","inputHash":"a127d7c20a4c9e444fb0638420f5c2dec7f1004d66df52c17a4eba736b718f2d","decidedAt":"2026-10-30T01:19:08.028Z"},"provenance":{"textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observationId":"7696880cb8e92f84076de13185aa87c5ebf806dd89f6282ee97bd5b6eca5a78a","retrievedAt":"2026-10-30T01:19:04.373Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"31fabc36-8701-40a1-8ae8-3776e7013344"},"reviewer":{"model":"gpt-6-luna","runId":"d4833719-d478-4193-88d3-5e80f8832def"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-06T21:46:28.332Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Ill, Hannes","url":null,"kind":"unknown"},{"name":"Attrach, Rafi Al","url":null,"kind":"unknown"},{"name":"Fani, Rajna","url":null,"kind":"unknown"},{"name":"Han, Ahram","url":null,"kind":"unknown"},{"name":"Yamamoto, Ryohei","url":null,"kind":"unknown"},{"name":"Moerig, Marlene Louisa","url":null,"kind":"unknown"},{"name":"Yoon, Dukyong","url":null,"kind":"unknown"},{"name":"Celi, Leo Anthony","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Ill, Hannes","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ill, Hannes","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37459","locator":"head/meta[name=author|citation_author]"},"assertionId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-06T22:50:09.637Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d","sourceUrl":"https://arxiv.org/abs/2610.37459","contentSha256":"e831354346460b9c7845a060c0a82678fb3499c3edd0bc6a98694dee152ba6b5","textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observedAt":"2026-11-06T22:50:09.637Z"}},{"actor":{"name":"Attrach, Rafi Al","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Attrach, Rafi Al","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37459","locator":"head/meta[name=author|citation_author]"},"assertionId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-06T22:50:09.637Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d","sourceUrl":"https://arxiv.org/abs/2610.37459","contentSha256":"e831354346460b9c7845a060c0a82678fb3499c3edd0bc6a98694dee152ba6b5","textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observedAt":"2026-11-06T22:50:09.637Z"}},{"actor":{"name":"Fani, Rajna","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Fani, Rajna","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37459","locator":"head/meta[name=author|citation_author]"},"assertionId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-06T22:50:09.637Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d","sourceUrl":"https://arxiv.org/abs/2610.37459","contentSha256":"e831354346460b9c7845a060c0a82678fb3499c3edd0bc6a98694dee152ba6b5","textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observedAt":"2026-11-06T22:50:09.637Z"}},{"actor":{"name":"Han, Ahram","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Han, Ahram","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37459","locator":"head/meta[name=author|citation_author]"},"assertionId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-06T22:50:09.637Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d","sourceUrl":"https://arxiv.org/abs/2610.37459","contentSha256":"e831354346460b9c7845a060c0a82678fb3499c3edd0bc6a98694dee152ba6b5","textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observedAt":"2026-11-06T22:50:09.637Z"}},{"actor":{"name":"Yamamoto, Ryohei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yamamoto, Ryohei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37459","locator":"head/meta[name=author|citation_author]"},"assertionId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-06T22:50:09.637Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d","sourceUrl":"https://arxiv.org/abs/2610.37459","contentSha256":"e831354346460b9c7845a060c0a82678fb3499c3edd0bc6a98694dee152ba6b5","textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observedAt":"2026-11-06T22:50:09.637Z"}},{"actor":{"name":"Moerig, Marlene Louisa","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Moerig, Marlene Louisa","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37459","locator":"head/meta[name=author|citation_author]"},"assertionId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-06T22:50:09.637Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d","sourceUrl":"https://arxiv.org/abs/2610.37459","contentSha256":"e831354346460b9c7845a060c0a82678fb3499c3edd0bc6a98694dee152ba6b5","textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observedAt":"2026-11-06T22:50:09.637Z"}},{"actor":{"name":"Yoon, Dukyong","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yoon, Dukyong","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37459","locator":"head/meta[name=author|citation_author]"},"assertionId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-06T22:50:09.637Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d","sourceUrl":"https://arxiv.org/abs/2610.37459","contentSha256":"e831354346460b9c7845a060c0a82678fb3499c3edd0bc6a98694dee152ba6b5","textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observedAt":"2026-11-06T22:50:09.637Z"}},{"actor":{"name":"Celi, Leo Anthony","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Celi, Leo Anthony","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37459","locator":"head/meta[name=author|citation_author]"},"assertionId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-06T22:50:09.637Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d","sourceUrl":"https://arxiv.org/abs/2610.37459","contentSha256":"e831354346460b9c7845a060c0a82678fb3499c3edd0bc6a98694dee152ba6b5","textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observedAt":"2026-11-06T22:50:09.637Z"}}],"participationCoverage":{"total":8,"returned":8,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.37459","snapshotId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d","contentSha256":"e831354346460b9c7845a060c0a82678fb3499c3edd0bc6a98694dee152ba6b5","textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observedAt":"2026-11-06T22:50:09.637Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/638c56be70b3c9795cff635aced0b4d62a06227910c80f26d810f420f2729d0c/source"}},"canonicalUrl":"https://escapemeridian.com/readings/ill-and-colleagues-test-how-ai-coding-agents-derive-clinical-638c56be","formats":{"html":"/readings/ill-and-colleagues-test-how-ai-coding-agents-derive-clinical-638c56be","markdown":"/readings/ill-and-colleagues-test-how-ai-coding-agents-derive-clinical-638c56be.md","json":"/readings/ill-and-colleagues-test-how-ai-coding-agents-derive-clinical-638c56be.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"},{"id":"time","name":"TIME","kind":"organization","context":"TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.","referenceUrls":["https://time.com/"],"path":"/entities/time","json":"/entities/time.json","markdown":"/entities/time.md"},{"id":"claude","name":"Claude","kind":"model","context":"Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.","referenceUrls":["https://claude.com/product/overview","https://www.anthropic.com/news","https://www.anthropic.com/research/team/economics"],"path":"/entities/claude","json":"/entities/claude.json","markdown":"/entities/claude.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"638c56be70b3c9795cff635aced0b4d62a06227910c80f26d810f420f2729d0c","revision":"c3071564-7547-4846-8ca1-c4c0056c4cf9","revisionKind":"publication-id","title":"Ill and colleagues test how AI coding agents derive clinical variables from EHR data","url":"https://escapemeridian.com/readings/ill-and-colleagues-test-how-ai-coding-agents-derive-clinical-638c56be"},"source":{"type":"source","id":"https://arxiv.org/abs/2610.37459","revision":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","revisionKind":"sha256-source-text","title":"M4Bench: Evaluating Procedural Specification for Clinical EHR Derivation Agents","url":"https://arxiv.org/abs/2610.37459"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"638c56be70b3c9795cff635aced0b4d62a06227910c80f26d810f420f2729d0c","title":"Ill and colleagues test how AI coding agents derive clinical variables from EHR data","argument":"Hannes Ill and coauthors introduce M4Bench, a benchmark for testing whether AI coding agents can reproduce clinical research variables derived from electronic health records. These variables depend on choices such as which item codes and time windows to use, how to handle missing data, and what each output row represents.\n\nIn a Docker-isolated Codex campaign, clinician-reviewed skills raised reward compared with no skills. The authors report that clearer, operational specifications accounted for most of the gain in a control subset; adding skills to those specifications contributed little. A descriptive Claude Code replication showed a similar direction of change.\n\nThe gains were partial agreement with reference results, not reliable task completion: few runs passed every strict check, and skills barely changed which rows agents returned. The benchmark therefore highlights a distinction between helping an agent follow a sufficiently precise procedure and getting it to return the intended data. How these results carry over to clinical research workflows beyond the tested datasets and systems remains unknown.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported results come from benchmark tasks on MIMIC-IV and eICU and specified coding-agent campaigns; the abstract does not establish performance in other clinical research workflows. The authors characterize the Claude Code replication as descriptive. The reported gains are partial agreement with reference results, not completed tasks.","speaker":{"name":"Ill, Hannes, Attrach, Rafi Al, Fani, Rajna, Han, Ahram, Yamamoto, Ryohei, Moerig, Marlene Louisa, Yoon, Dukyong, Celi, Leo Anthony","roleAtSource":"Article author"},"source":{"title":"M4Bench: Evaluating Procedural Specification for Clinical EHR Derivation Agents","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.37459","publishedAt":"2026-10-27","dateBasis":"Date basis not provided."},"evidence":[{"text":"Ill and coauthors introduce M4Bench, a benchmark of clinical derivation tasks on MIMIC-IV and eICU, to evaluate AI coding agents and the effect of clinician-reviewed skills.","excerpt":"We introduce M4Bench, 28 clinical derivation tasks across 15 task families on MIMIC-IV and eICU","url":"https://arxiv.org/abs/2610.37459","relationship":"Source-stated"},{"text":"In the Codex campaign, skills raised reward over the no-skill baseline, but the authors attribute most of the measured gain to specification completeness.","excerpt":"skills raise reward by 0.136 over the no-skill baseline, with 26 of 28 tasks improving. Specification completeness accounts for most of this gain","url":"https://arxiv.org/abs/2610.37459","relationship":"Source-stated"},{"text":"The authors report that skills barely changed which rows agents returned and that few runs passed every strict check.","excerpt":"only 25 of 280 with-skill and 13 of 280 no-skill Codex runs pass every strict check, and skills barely change which rows agents return (key F1 +0.003).","url":"https://arxiv.org/abs/2610.37459","relationship":"Source-stated"},{"text":"The abstract describes a directionally consistent Claude Code replication, while characterizing it as descriptive.","excerpt":"A descriptive Claude Code replication on all 28 tasks (Opus 4.7 and Sonnet 4.6, 560 runs) is directionally consistent","url":"https://arxiv.org/abs/2610.37459","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.37459","checkedAt":"2026-11-06T22:50:09.637Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Would clearer specifications improve strict-check completion in clinical research workflows beyond the datasets and systems tested?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Ill and colleagues test how AI coding agents derive clinical variables from EHR data","summary":"Hannes Ill and coauthors introduce M4Bench, a benchmark for testing whether AI coding agents can reproduce clinical research variables derived from electronic health records. These variables depend on choices such as which item codes and time windows to use, how to handle missing data, and what each output row represents.\n\nIn a Docker-isolated Codex campaign, clinician-reviewed skills raised reward compared with no skills. The authors report that clearer, operational specifications accounted for most of the gain in a control subset; adding skills to those specifications contributed little. A descriptive Claude Code replication showed a similar direction of change.\n\nThe gains were partial agreement with reference results, not reliable task completion: few runs passed every strict check, and skills barely changed which rows agents returned. The benchmark therefore highlights a distinction between helping an agent follow a sufficiently precise procedure and getting it to return the intended data. How these results carry over to clinical research workflows beyond the tested datasets and systems remains unknown.","question":"Would clearer specifications improve strict-check completion in clinical research workflows beyond the datasets and systems tested?","claims":[{"text":"Ill and coauthors introduce M4Bench, a benchmark of clinical derivation tasks on MIMIC-IV and eICU, to evaluate AI coding agents and the effect of clinician-reviewed skills.","excerpt":"We introduce M4Bench, 28 clinical derivation tasks across 15 task families on MIMIC-IV and eICU","support":"source-stated"},{"text":"In the Codex campaign, skills raised reward over the no-skill baseline, but the authors attribute most of the measured gain to specification completeness.","excerpt":"skills raise reward by 0.136 over the no-skill baseline, with 26 of 28 tasks improving. Specification completeness accounts for most of this gain","support":"source-stated"},{"text":"The authors report that skills barely changed which rows agents returned and that few runs passed every strict check.","excerpt":"only 25 of 280 with-skill and 13 of 280 no-skill Codex runs pass every strict check, and skills barely change which rows agents return (key F1 +0.003).","support":"source-stated"},{"text":"The abstract describes a directionally consistent Claude Code replication, while characterizing it as descriptive.","excerpt":"A descriptive Claude Code replication on all 28 tasks (Opus 4.7 and Sonnet 4.6, 560 runs) is directionally consistent","support":"source-stated"}],"limitations":["The reported results come from benchmark tasks on MIMIC-IV and eICU and specified coding-agent campaigns; the abstract does not establish performance in other clinical research workflows.","The authors characterize the Claude Code replication as descriptive.","The reported gains are partial agreement with reference results, not completed tasks."],"mediaId":"fec5f639cc2380fb76770a155f83d6a1753a8b1ca8c4049ccdfecdd210dc6961","correctionReason":null,"id":"638c56be70b3c9795cff635aced0b4d62a06227910c80f26d810f420f2729d0c","publicationId":"c3071564-7547-4846-8ca1-c4c0056c4cf9","version":1,"sourceUrl":"https://arxiv.org/abs/2610.37459","sourceTitle":"M4Bench: Evaluating Procedural Specification for Clinical EHR Derivation Agents","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-27","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-28T05:54:32.814Z","createdAt":"2026-11-06T21:46:28.332Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"fec5f639cc2380fb76770a155f83d6a1753a8b1ca8c4049ccdfecdd210dc6961","kind":"article","url":"https://arxiv.org/abs/2610.37459","sourceUrl":"https://arxiv.org/abs/2610.37459","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-30T01:19:04.866Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.37459","checkedAt":"2026-11-06T22:50:09.637Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"fec5f639cc2380fb76770a155f83d6a1753a8b1ca8c4049ccdfecdd210dc6961","confidence":0.83,"provider":"typesafe","model":"jev-1.13.0","inputHash":"a127d7c20a4c9e444fb0638420f5c2dec7f1004d66df52c17a4eba736b718f2d","decidedAt":"2026-10-30T01:19:08.028Z"},"provenance":{"textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observationId":"7696880cb8e92f84076de13185aa87c5ebf806dd89f6282ee97bd5b6eca5a78a","retrievedAt":"2026-10-30T01:19:04.373Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"31fabc36-8701-40a1-8ae8-3776e7013344"},"reviewer":{"model":"gpt-6-luna","runId":"d4833719-d478-4193-88d3-5e80f8832def"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-06T21:46:28.332Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Ill, Hannes","url":null,"kind":"unknown"},{"name":"Attrach, Rafi Al","url":null,"kind":"unknown"},{"name":"Fani, Rajna","url":null,"kind":"unknown"},{"name":"Han, Ahram","url":null,"kind":"unknown"},{"name":"Yamamoto, Ryohei","url":null,"kind":"unknown"},{"name":"Moerig, Marlene Louisa","url":null,"kind":"unknown"},{"name":"Yoon, Dukyong","url":null,"kind":"unknown"},{"name":"Celi, Leo Anthony","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Ill, Hannes","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ill, Hannes","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37459","locator":"head/meta[name=author|citation_author]"},"assertionId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-06T22:50:09.637Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d","sourceUrl":"https://arxiv.org/abs/2610.37459","contentSha256":"e831354346460b9c7845a060c0a82678fb3499c3edd0bc6a98694dee152ba6b5","textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observedAt":"2026-11-06T22:50:09.637Z"}},{"actor":{"name":"Attrach, Rafi Al","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Attrach, Rafi Al","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37459","locator":"head/meta[name=author|citation_author]"},"assertionId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-06T22:50:09.637Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d","sourceUrl":"https://arxiv.org/abs/2610.37459","contentSha256":"e831354346460b9c7845a060c0a82678fb3499c3edd0bc6a98694dee152ba6b5","textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observedAt":"2026-11-06T22:50:09.637Z"}},{"actor":{"name":"Fani, Rajna","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Fani, Rajna","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37459","locator":"head/meta[name=author|citation_author]"},"assertionId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-06T22:50:09.637Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d","sourceUrl":"https://arxiv.org/abs/2610.37459","contentSha256":"e831354346460b9c7845a060c0a82678fb3499c3edd0bc6a98694dee152ba6b5","textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observedAt":"2026-11-06T22:50:09.637Z"}},{"actor":{"name":"Han, Ahram","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Han, Ahram","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37459","locator":"head/meta[name=author|citation_author]"},"assertionId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-06T22:50:09.637Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d","sourceUrl":"https://arxiv.org/abs/2610.37459","contentSha256":"e831354346460b9c7845a060c0a82678fb3499c3edd0bc6a98694dee152ba6b5","textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observedAt":"2026-11-06T22:50:09.637Z"}},{"actor":{"name":"Yamamoto, Ryohei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yamamoto, Ryohei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37459","locator":"head/meta[name=author|citation_author]"},"assertionId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-06T22:50:09.637Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d","sourceUrl":"https://arxiv.org/abs/2610.37459","contentSha256":"e831354346460b9c7845a060c0a82678fb3499c3edd0bc6a98694dee152ba6b5","textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observedAt":"2026-11-06T22:50:09.637Z"}},{"actor":{"name":"Moerig, Marlene Louisa","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Moerig, Marlene Louisa","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37459","locator":"head/meta[name=author|citation_author]"},"assertionId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-06T22:50:09.637Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d","sourceUrl":"https://arxiv.org/abs/2610.37459","contentSha256":"e831354346460b9c7845a060c0a82678fb3499c3edd0bc6a98694dee152ba6b5","textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observedAt":"2026-11-06T22:50:09.637Z"}},{"actor":{"name":"Yoon, Dukyong","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yoon, Dukyong","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37459","locator":"head/meta[name=author|citation_author]"},"assertionId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-06T22:50:09.637Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d","sourceUrl":"https://arxiv.org/abs/2610.37459","contentSha256":"e831354346460b9c7845a060c0a82678fb3499c3edd0bc6a98694dee152ba6b5","textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observedAt":"2026-11-06T22:50:09.637Z"}},{"actor":{"name":"Celi, Leo Anthony","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Celi, Leo Anthony","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37459","locator":"head/meta[name=author|citation_author]"},"assertionId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-06T22:50:09.637Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d","sourceUrl":"https://arxiv.org/abs/2610.37459","contentSha256":"e831354346460b9c7845a060c0a82678fb3499c3edd0bc6a98694dee152ba6b5","textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observedAt":"2026-11-06T22:50:09.637Z"}}],"participationCoverage":{"total":8,"returned":8,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.37459","snapshotId":"72cea77c747bf16c41818fbf1ad3ca4b19203660d832f07dc5d41bafef3b1c2d","contentSha256":"e831354346460b9c7845a060c0a82678fb3499c3edd0bc6a98694dee152ba6b5","textHash":"d572a36b6d0d206db9c53470ee07fcd28284d6c7427882fe63dc69eeb122e4ac","observedAt":"2026-11-06T22:50:09.637Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/638c56be70b3c9795cff635aced0b4d62a06227910c80f26d810f420f2729d0c/source"}}}],"audits":[{"publicationId":"c3071564-7547-4846-8ca1-c4c0056c4cf9","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The draft correctly credits Hannes Ill and coauthors and keeps the findings attributed to the authors, consistent with the arXiv author list and abstract.","excerpt":"Authors:Hannes Ill, Rafi Al Attrach, Rajna Fani, Ahram Han, Ryohei Yamamoto, Marlene Louisa Moerig, Dukyong Yoon, Leo Anthony Celi"},{"dimension":"summary","status":"uncertain","explanation":"The draft explains derived variables well, but leaves central terms unclear. \"reward\" is the core metric and is never defined. \"skills\" (clinician-reviewed markdown instructions given to the agent) is also unexplained, as are \"Docker-isolated Codex campaign\" and MIMIC-IV/eICU, so a non-specialist cannot tell what improved. \"clearer, operational specifications\" paraphrases the technical term \"specification completeness\"; restore that term. Smallest repair: briefly gloss reward as partial agreement with a reference result, skills as reviewed markdown instructions, and MIMIC-IV/eICU as clinical databases.","excerpt":"Skills are released as a versioned, clinician-reviewable markdown delivery format"},{"dimension":"date","status":"pass","explanation":"originalPublished 2026-10-27 matches the arXiv v1 submission date. The later feed date is correctly not used.","excerpt":"[Submitted on 27 Oct 2026]"},{"dimension":"support","status":"uncertain","explanation":"Most claims match the abstract. However, the sentence saying the benchmark \"therefore highlights a distinction between helping an agent follow a sufficiently precise procedure and getting it to return the intended data\" is an unlabeled inference. The abstract reports row-level change (key F1) only for skills, not for operational specs alone, so the distinction is not directly established. Smallest repair: label the sentence as Escape Meridian interpretation and make it conditional. The question is open and acceptable, although the abstract reports spec effects on reward, not on strict-check pass rates.","excerpt":"skills barely change which rows agents return (key F1 +0.003)"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified arXiv abstract page for this paper, the direct original source.","excerpt":null}],"createdAt":"2026-11-06T21:46:28.332Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}