Koyun, Alicem · Nov 4, 2026
Does Your Trial Simulator Know What the Drug Does? An Intervention-Blind Test of Interventional Validity on 7,207 Registered Trials
Markdown · JSON · Original source
Alicem Koyun argues that trial simulators should be tested on whether they use information about the intervention—the treatment being tested—not just on whether they predict trial outcomes. The paper measures this as interventional skill: how much prediction error a full model removes compared with the same model when intervention-identifying fields are withheld. Across a temporal split of registered trials, the registry-trained simulator’s interventional skill is negative: withholding intervention information made it perform better. A frontier language model given only the registered protocol had positive interventional skill, while smaller and were at the limit of detection. In perturbation tests, swapping the treatment arms reversed the large model’s prediction in most contrasts, which Koyun presents as evidence that it uses the intervention as such. But the comparison is not fully drug-blind: a language model could recover the withheld drug class from the blind prompt 45% of the . Koyun therefore describes the measured interventional skill as a conservative lower bound. The paper also reports that calibrated error control did not transfer under temporal shift. The , code, prompts and per-call model outputs are released.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 4, 2026 (2026-11-04).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T03:02:25.068Z. First observed: 2026-11-05T05:06:26.771Z. Retrieved: 2026-11-10T02:55:47.057Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The intervention-blind reference was not fully drug-blind: a language model recovered the withheld drug class from the blind prompt 45% of the , so the paper calls interventional skill a conservative lower bound. The reported evaluation includes a temporal test split of 878 contrasts from 655 trials; the supplied evidence does not establish how results generalize beyond that evaluation. The paper reports that calibrated error control did not transfer under temporal shift; the supplied evidence does not establish whether the frontier language model’s positive interventional skill persists under such a shift.
Open question
Would the frontier language model retain positive interventional skill under temporal shift, given that calibrated error control did not transfer?
Source-stated
Koyun proposes measuring whether a trial simulator’s accuracy depends on knowing which intervention was tested, by comparing it with the same model while intervention-identifying fields are withheld.
we score a simulator on how much of its accuracy depends on knowing which intervention was tested.
Source-stated
In the temporal test split, the registry-trained simulator had negative interventional skill; withholding intervention information made it perform better.
its interventional skill is -0.12 (95% CI -0.21 to -0.05): withholding the intervention makes it better
Source-stated
The frontier language model reading only the registered protocol had positive interventional skill; smaller and open-weight models were at the limit of detection.
A frontier language model reading only the registered protocol has interventional skill 0.20 [0.08, 0.31]; smaller and open-weight models sit at the limit of detection.
Source-stated
Swapping treatment arms reversed the large language model’s prediction in most contrasts, which the paper offers as evidence that it uses the intervention as such.
swapping the arms reverses the sign of its prediction in 86% of contrasts.
Source-stated
The intervention-blind reference could still reveal drug-class information, so the paper characterizes interventional skill as a conservative lower bound.
a language model recovers the withheld drug class 45% of the time from the blind prompt
Source-stated
The paper reports that calibrated error control did not transfer under temporal shift.
calibrated error control does not transfer under temporal shift.
Source and contributor context
Koyun, Alicem · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Open weights
A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: c91eb668-8f39-4d94-86f2-46d781735d01.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T03:02:25.068Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 80b53e42d0c6719409b57efb9818de0915384e32972dc93f2a7f5af734a5dbde. Source text hash: dc4e4366773949071a55956bb84fa990661bc4a6c993429770d02db34a259168.
Revision and source-check history
Version 1 · 2026-11-10T03:02:25.068Z
Does Your Trial Simulator Know What the Drug Does? An Intervention-Blind Test of Interventional Validity on 7,207 Registered Trials
Alicem Koyun argues that trial simulators should be tested on whether they use information about the intervention—the treatment being tested—not just on whether they predict trial outcomes. The paper measures this as interventional skill: how much prediction error a full model removes compared with the same model when intervention-identifying fields are withheld. Across a temporal split of registered trials, the registry-trained simulator’s interventional skill is negative: withholding intervention information made it perform better. A frontier language model given only the registered protocol had positive interventional skill, while smaller and were at the limit of detection. In perturbation tests, swapping the treatment arms reversed the large model’s prediction in most contrasts, which Koyun presents as evidence that it uses the intervention as such. But the comparison is not fully drug-blind: a language model could recover the withheld drug class from the blind prompt 45% of the . Koyun therefore describes the measured interventional skill as a conservative lower bound. The paper also reports that calibrated error control did not transfer under temporal shift. The , code, prompts and per-call model outputs are released.
Original source
cs.AI · Article
Source details
Original source date: Nov 4, 2026 (2026-11-04).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T03:02:25.068Z. First observed: 2026-11-05T05:06:26.771Z. Retrieved: 2026-11-10T02:55:47.057Z.
Limits
The intervention-blind reference was not fully drug-blind: a language model recovered the withheld drug class from the blind prompt 45% of the , so the paper calls interventional skill a conservative lower bound. The reported evaluation includes a temporal test split of 878 contrasts from 655 trials; the supplied evidence does not establish how results generalize beyond that evaluation. The paper reports that calibrated error control did not transfer under temporal shift; the supplied evidence does not establish whether the frontier language model’s positive interventional skill persists under such a shift.
Open question
Would the frontier language model retain positive interventional skill under temporal shift, given that calibrated error control did not transfer?
Source-stated
Koyun proposes measuring whether a trial simulator’s accuracy depends on knowing which intervention was tested, by comparing it with the same model while intervention-identifying fields are withheld.
we score a simulator on how much of its accuracy depends on knowing which intervention was tested.
Source-stated
In the temporal test split, the registry-trained simulator had negative interventional skill; withholding intervention information made it perform better.
its interventional skill is -0.12 (95% CI -0.21 to -0.05): withholding the intervention makes it better
Source-stated
The frontier language model reading only the registered protocol had positive interventional skill; smaller and open-weight models were at the limit of detection.
A frontier language model reading only the registered protocol has interventional skill 0.20 [0.08, 0.31]; smaller and open-weight models sit at the limit of detection.
Source-stated
Swapping treatment arms reversed the large language model’s prediction in most contrasts, which the paper offers as evidence that it uses the intervention as such.
swapping the arms reverses the sign of its prediction in 86% of contrasts.
Source-stated
The intervention-blind reference could still reveal drug-class information, so the paper characterizes interventional skill as a conservative lower bound.
a language model recovers the withheld drug class 45% of the time from the blind prompt
Source-stated
The paper reports that calibrated error control did not transfer under temporal shift.
calibrated error control does not transfer under temporal shift.
Source and contributor context
Koyun, Alicem · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Open weights
A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: c91eb668-8f39-4d94-86f2-46d781735d01.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T03:02:25.068Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 80b53e42d0c6719409b57efb9818de0915384e32972dc93f2a7f5af734a5dbde. Source text hash: dc4e4366773949071a55956bb84fa990661bc4a6c993429770d02db34a259168.
uncertain · 2026-11-10T03:02:25.068Z · anthropic/claude-opus-5.5
Publication: c91eb668-8f39-4d94-86f2-46d781735d01
attribution — pass: The paper is correctly attributed to its sole listed author, Alicem Koyun. Its findings are framed as the paper's reported results rather than as established facts.
Authors:Alicem Koyun
summary — uncertain: The reading drops the paper's central tension: the registry simulator removes a fifth of a trivial baseline's error, yet it has negative interventional skill. In other words, it looks accurate without using what the drug does. The framing that ladder positions are assigned by reading papers, not by measurement, is also missing. Several central terms are unexplained for lay readers: trial simulator/patient world model, temporal split, frontier vs open-weight models, and perturbation test. The smallest repair is to add the one-fifth error result and plain-language glosses for these terms.
a registry-trained gradient-boosting simulator removes a fifth of the error of a trivial baseline, but its interventional skill is -0.12
date — pass: originalPublished is 2026-11-04, which matches the arXiv v1 submission date. The feed date of 2026-11-05 is later and correctly not used.
[Submitted on 4 Nov 2026]
support — uncertain: The question and the third limitation link the reliability layer's failure to transfer calibrated error control to whether the language model's interventional skill persists under temporal shift. The abstract reports the language model's skill right after the temporal-split result, which suggests it may already be measured under that split. Calibrated error control is a separate reliability-layer finding. The summary also omits that ranking by predicted error does lower error. The repair is to reframe the question around the reliability-layer result, or to state that the abstract is ambiguous about the language model's test split.
A reliability layer shows that ranking by predicted error lowers it but that calibrated error control does not transfer under temporal shift.
media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEA model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIME