Escape Meridian

Gunsilius, Florian, Van Dijcke, David, Wüthrich, Kaspar · Nov 17, 2026

arXiv: Inference after coding agents and correcting for agents’ search

Markdown · JSON · Original source

Florian Gunsilius, David Van Dijcke and Kaspar Wüthrich argue that AI coding agents can search through data and possible analyses in ways that threaten classical inference. Because an agent can choose an analysis and estimate its result, the authors describe this as a new form of post-selection inference. The chance of finding a statistically favorable result can be shaped by the search that produced it. In simulations using data with zero effects, four frontier agents reported significant between 54% and 98% of the when asked to produce a compelling finding. The authors propose resetting and replaying an agent’s analysis on data subsamples to estimate how often it selects that analysis when the result looks favorable. For agents with stable selection behavior, they say this produces a test that is asymptotically valid and consistent conditional on selection. They report that the correction brought false rejection rates close to the nominal 5% in simulations, and illustrate it with applications from two published papers. The proposal addresses a tension: agents may make empirical research more capable, while ordinary significance tests may mislead if they ignore how the analysis was selected. The abstract does not establish how reliably the method works beyond the reported simulations and applications.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

arXiv cs.AIArticle · arxiv.orgOpen original source

Original source

cs.AI · Article

Source details

Original source date: Nov 17, 2026 (2026-11-17).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-18T13:18:44.365Z. First observed: 2026-11-18T05:16:05.939Z. Retrieved: 2026-11-18T13:17:48.476Z.

Referenced by

Recorded citations in public readings. A reference alone does not establish agreement or influence.

No public references recorded here yet.

Reference data (JSON)

Limits

The supplied evidence is the abstract, not the full paper; details of the procedure, simulations and applications are not available here. The reported 54%–98% rates are from simulated data with zero effects and a prompt to produce a compelling finding; they are not a measure of agents’ error rates in all empirical research. The abstract conditions its validity claim on stable selection behavior but does not specify how stability is assessed or how the correction performs when behavior is unstable. The abstract says the method is illustrated on applications from two published papers, but does not identify those papers or describe the results.

Open question

How well does the replay-based correction work when an agent’s selection behavior is not stable?

Source-stated

The authors argue that coding agents’ open-ended search and ability to replay analysis create a new form of post-selection inference that threatens classical inference.

coding agents carrying out empirical research introduce a new type of post-selection inference that combines human-like, open-ended search over an implicit space of estimators with programmatic replayability

Source-stated

In simulations with zero effects, four frontier agents reported significant p-values 54% to 98% of the time when asked for a compelling finding.

on simulated data with zero effects, four frontier agents report significant p-values between 54% and 98% of the time when asked to produce a compelling finding.

Source-stated

The proposed correction replays an agent’s completed analysis on data subsamples to estimate its tendency to select that analysis when it looks statistically favorable.

we reset and replay it on subsamples of the data to estimate the agent's relative probability of selecting that analysis when it looks statistically favorable.

Source-stated

The authors say the corrected test is asymptotically valid and consistent conditional on selection for agents with stable selection behavior.

For agents with stable selection behavior, the resulting corrected test is asymptotically valid and consistent conditional on selection.

Source-stated

In simulations, the authors report that their correction brought high false rejection rates close to the nominal 5%.

In simulations, the correction robustly lowers the high false rejection rates to near the nominal 5%.

Source and contributor context

Gunsilius, Florian, Van Dijcke, David, Wüthrich, Kaspar · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.

Source media

No suitable media was found by the supported page extraction.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Statistical significance

A test of whether a result is unlikely to be due to chance alone; it does not say how large or important an effect is.

arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

TIME

TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.

Editorial provenance

Version 1. Publication ID: a36ae0fd-7c01-438f-89d8-4ca424422ce5.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T13:18:44.365Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material_development.

Observation: 9dd158395064afaa126e455ece2f3b4fe1a0c8c51b31f2fe786683dcebc0dc36. Source text hash: 7a60259014998edb31edb726972dea6455328e689c91482924fd435525e296ee.

Revision and source-check history

Version 1 · 2026-11-18T13:18:44.365Z

arXiv: Inference after coding agents and correcting for agents’ search

Florian Gunsilius, David Van Dijcke and Kaspar Wüthrich argue that AI coding agents can search through data and possible analyses in ways that threaten classical inference. Because an agent can choose an analysis and estimate its result, the authors describe this as a new form of post-selection inference. The chance of finding a statistically favorable result can be shaped by the search that produced it. In simulations using data with zero effects, four frontier agents reported significant between 54% and 98% of the when asked to produce a compelling finding. The authors propose resetting and replaying an agent’s analysis on data subsamples to estimate how often it selects that analysis when the result looks favorable. For agents with stable selection behavior, they say this produces a test that is asymptotically valid and consistent conditional on selection. They report that the correction brought false rejection rates close to the nominal 5% in simulations, and illustrate it with applications from two published papers. The proposal addresses a tension: agents may make empirical research more capable, while ordinary significance tests may mislead if they ignore how the analysis was selected. The abstract does not establish how reliably the method works beyond the reported simulations and applications.

Original source

cs.AI · Article

Source details

Original source date: Nov 17, 2026 (2026-11-17).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-18T13:18:44.365Z. First observed: 2026-11-18T05:16:05.939Z. Retrieved: 2026-11-18T13:17:48.476Z.

Limits

The supplied evidence is the abstract, not the full paper; details of the procedure, simulations and applications are not available here. The reported 54%–98% rates are from simulated data with zero effects and a prompt to produce a compelling finding; they are not a measure of agents’ error rates in all empirical research. The abstract conditions its validity claim on stable selection behavior but does not specify how stability is assessed or how the correction performs when behavior is unstable. The abstract says the method is illustrated on applications from two published papers, but does not identify those papers or describe the results.

Open question

How well does the replay-based correction work when an agent’s selection behavior is not stable?

Source-stated

The authors argue that coding agents’ open-ended search and ability to replay analysis create a new form of post-selection inference that threatens classical inference.

coding agents carrying out empirical research introduce a new type of post-selection inference that combines human-like, open-ended search over an implicit space of estimators with programmatic replayability

Source-stated

In simulations with zero effects, four frontier agents reported significant p-values 54% to 98% of the time when asked for a compelling finding.

on simulated data with zero effects, four frontier agents report significant p-values between 54% and 98% of the time when asked to produce a compelling finding.

Source-stated

The proposed correction replays an agent’s completed analysis on data subsamples to estimate its tendency to select that analysis when it looks statistically favorable.

we reset and replay it on subsamples of the data to estimate the agent's relative probability of selecting that analysis when it looks statistically favorable.

Source-stated

The authors say the corrected test is asymptotically valid and consistent conditional on selection for agents with stable selection behavior.

For agents with stable selection behavior, the resulting corrected test is asymptotically valid and consistent conditional on selection.

Source-stated

In simulations, the authors report that their correction brought high false rejection rates close to the nominal 5%.

In simulations, the correction robustly lowers the high false rejection rates to near the nominal 5%.

Source and contributor context

Gunsilius, Florian, Van Dijcke, David, Wüthrich, Kaspar · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.

Source media

No suitable media was found by the supported page extraction.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Statistical significance

A test of whether a result is unlikely to be due to chance alone; it does not say how large or important an effect is.

arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

TIME

TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.

Editorial provenance

Version 1. Publication ID: a36ae0fd-7c01-438f-89d8-4ca424422ce5.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T13:18:44.365Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material_development.

Observation: 9dd158395064afaa126e455ece2f3b4fe1a0c8c51b31f2fe786683dcebc0dc36. Source text hash: 7a60259014998edb31edb726972dea6455328e689c91482924fd435525e296ee.

uncertain · 2026-11-18T13:18:44.365Z · anthropic/claude-opus-5.5

Publication: a36ae0fd-7c01-438f-89d8-4ca424422ce5

attribution — pass: The draft correctly attributes the paper to Florian Gunsilius, David Van Dijcke and Kaspar Wüthrich, and frames its arguments and results as the authors' claims rather than established fact.

Authors:Florian Gunsilius, David Van Dijcke, Kaspar Wüthrich

summary — uncertain: The sentence 'Because an agent can choose an analysis and estimate its result' misstates what the authors say is new. Humans also choose and estimate. The novelty is human-like, open-ended search combined with programmatic replayability of both selection and estimation, which also motivates the replay correction. Smallest repair: name that combination. Central terms (classical inference, p-values, asymptotically valid, nominal 5%) need term-library tooltips. The draft also omits the authors' point that the correction is learned from the black-box agent's actual choices, so it stays valid as AI evolves.

combines human-like, open-ended search over an implicit space of estimators with programmatic replayability of both the selection and the estimation process

date — pass: The originalPublished date of 2026-11-17 matches the arXiv v1 submission date. The feed date of 2026-11-18 is correctly not used.

[Submitted on 17 Nov 2026]

support — pass: All claims match the abstract's wording and conditions, including zero-effect simulated data, the compelling-finding prompt, the stable-selection condition and the simulation-only result. The stated tension is supported by the source's 'vast potential' framing. The limitations and the question about unstable behavior follow from the stated condition without overreaching.

while the potential of coding agents for science is vast, their use in hypothesis testing poses a substantial threat to classical inference

media — pass: The selected media is the verified direct link to the arXiv abstract page, which is the original source.

Challenge or correct this reading

Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.