Causal Discovery Bench: Can LLMs Autonomously Conduct Effective Experiments?

Causal Discovery Bench: Can LLMs Autonomously Conduct Effective Experiments?

In their paper, Zhuoyang Liu, Ning Peng, Ming Shen and Zijun Gao introduce Causal-Discovery-Bench, a sandbox for testing whether LLM agents can design experiments and discover causal relationships. The authors argue that evaluating agents only on reasoning from prepared evidence misses a separate challenge: choosing informative experiments, managing resources, interpreting observations as they arrive, and deciding when to stop. Across 3,802 episodes with 13 models from 5 vendors, they report a hierarchy of difficulty, from designing a single experiment to resolving conflicting evidence autonomously. Their analysis attributes 94.5% of agent mistakes to experimental design rather than inference. They also report that relaxing resource constraints alone does not reliably fix failures in sequential discovery, while basic statistical tools improve performance across multiple model families. Advanced tools add little when models lack prerequisite lower-level capabilities. The results point to experiment design—not just drawing conclusions from curated evidence—as a central part of autonomous research, within the benchmark’s configurable sandbox.

Publication: 997bcdeb-4b63-4aca-8524-c1c9b14e3c52

attribution — pass: The reading correctly credits the paper and its findings to Zhuoyang Liu, Ning Peng, Ming Shen and Zijun Gao. Results are framed as what the authors introduce, report or attribute, not as established fact.

Authors:Zhuoyang Liu, Ning Peng, Ming Shen, Zijun Gao

summary — pass: The reading leads with the actor and the development: a benchmark testing whether LLM agents can run experiments. It explains the skills involved in plain language, and the closing phrase 'drawing conclusions from curated evidence' clarifies what inference means here. It keeps the key results and scope: the four-level hierarchy, the 94.5% figure, the effect of resource constraints and the effect of tools. The title matches the source. The question is a genuine open issue about generalization. Terms such as 'sandbox' and 'LLM agents' could use term-library tooltips, but they do not block understanding.

they must not only reason over existing evidence, but also design informative experiments, allocate resources, analyze sequential observations, and decide when to stop and make conclusions

date — pass: originalPublished is 2026-10-13. That matches the v1 submission date on the arXiv page. The later feed date of 2026-11-10 is correctly not used.

[v1] Tue, 13 Oct 2026 15:54:50 UTC (2,253 KB)

support — pass: Each claim matches the abstract, including the episode, model and vendor counts, the 94.5% attribution, the resource-constraint finding and the tool findings. The closing interpretation is limited to the benchmark's sandbox and follows from the stated performance gap. The limitations correctly note that only the abstract is available and that generalization is untested. No unsupported benefits or costs are implied.

We identify a substantial performance gap between causal inference from correctly curated evidence and navigating causal discovery through autonomous experimentation

media — pass: The selected media is the verified arXiv abstract page. It is the direct original source and the most relevant access point. No other source media was found.