# arXiv cs.AI: Do Language Models Know What Compelling Scientific Evidence Looks Like? [Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-do-language-models-know-what-compelling-scientific-38ca62e3) The authors introduce EDB-Bio, a benchmark built from data in three biological domains, to test how language models assess evidence from multiple studies. In each of its 96k episodes, a model sees a sequence of experimental results and predicts the probability that a target hypothesis is true. They report that open-weight and frontier models, without additional training, show systematic underconfidence and achieve Brier scores comparable to a trivial base-rate forecaster. Post-training against strictly proper scoring rules improves calibration and discrimination. In a sequential experiment selection task, their Ibis0 model is reported to be 2.5 times more efficient than the best frontier model. The authors also use Bayes factors to examine how models update beliefs, whether their judgments about evidence relevance agree with expert biologists, and where inconsistencies and biases appear. Their results concern the benchmark and experiment selection task described in the abstract; the abstract does not establish how the findings transfer to other scientific domains or real-world discovery. The authors say they release the code, data, and trained models. What conditions and efficiency measure underlie the reported comparison between Ibis0 and the best frontier model in the sequential experiment selection task? Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Oct 29, 2026 (2026-10-29). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-10T11:34:16.028Z. First observed: 2026-11-02T05:01:23.175Z. Retrieved: 2026-11-02T10:46:17.526Z. - [Do Language Models Know What Compelling Scientific Evidence Looks Like?](https://arxiv.org/abs/2610.41015) ## Limits The available evidence is the paper's abstract; it does not give the methods or conditions needed to assess the reported 2.5-times efficiency comparison. The abstract describes three biological domains and a sequential experiment selection task, but does not establish how the reported results transfer to other scientific domains or real-world discovery. ## Open question What conditions and efficiency measure underlie the reported comparison between Ibis0 and the best frontier model in the sequential experiment selection task? ## Source-stated The authors introduce EDB-Bio, a benchmark based on real-world data from three biological domains, with episodes asking models to estimate the probability of a target hypothesis after seeing experimental results. > Each of EDB-Bio's 96k episodes presents a sequence of experimental results from published studies and asks the model to predict the probability that a target hypothesis is true. - [Supporting source](https://arxiv.org/abs/2610.41015) ## Source-stated The authors report systematic underconfidence and Brier scores comparable to a trivial base-rate forecaster for open-weight and frontier models without additional training. > Out-of-the-box, open-weight and frontier models alike demonstrate systematic underconfidence and attain Brier scores comparable to a trivial base-rate forecaster. - [Supporting source](https://arxiv.org/abs/2610.41015) ## Source-stated The authors report that post-training against strictly proper scoring rules improves calibration and discrimination, and that Ibis0 is more efficient in their sequential experiment selection task. > Post-training against strictly proper scoring rules significantly improves both calibration and discrimination. This calibration pays off downstream: in a sequential experiment selection task, our Ibis0 model is 2.5 times more efficient than the best frontier model. - [Supporting source](https://arxiv.org/abs/2610.41015) ## Source-stated The authors use Bayes factors to analyze model belief updates, agreement with expert biologists about evidence relevance, and inconsistencies and biases. > quantifying evidence weights as Bayes factors reveals how models update their beliefs over time, whether they agree with expert biologists on relevance, and where they exhibit inconsistencies and biases. - [Supporting source](https://arxiv.org/abs/2610.41015) ## Source and contributor context Igoe, Conor, Grand, Gabriel, Koch, Zane, Bou, Albert, Moss, Eli, Hinks, Michaela M., Rodriques, Samuel G., White, Andrew, Narayanan, Siddharth · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Open weights A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted. - [Definition reference](https://opensource.org/ai/open-weights) ## Frontier model One of the most capable AI models available at a given time; the set shifts as models improve. - [Definition reference](https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 9d580cff-87c7-432e-835a-039d2a0b7f84. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T11:34:16.028Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 202bf11c950fa49e01879d52c239c78b93c5ec95c07ddbf87d7ff4d848e14014. Source text hash: a6313d2ad3e2e0fedad891eb8c6e067b8282672f47a41d0809918e0234fd08aa. ## Revision and source-check history ### Version 1 — 2026-11-10T11:34:16.028Z The authors introduce EDB-Bio, a benchmark built from data in three biological domains, to test how language models assess evidence from multiple studies. In each of its 96k episodes, a model sees a sequence of experimental results and predicts the probability that a target hypothesis is true. They report that open-weight and frontier models, without additional training, show systematic underconfidence and achieve Brier scores comparable to a trivial base-rate forecaster. Post-training against strictly proper scoring rules improves calibration and discrimination. In a sequential experiment selection task, their Ibis0 model is reported to be 2.5 times more efficient than the best frontier model. The authors also use Bayes factors to examine how models update beliefs, whether their judgments about evidence relevance agree with expert biologists, and where inconsistencies and biases appear. Their results concern the benchmark and experiment selection task described in the abstract; the abstract does not establish how the findings transfer to other scientific domains or real-world discovery. The authors say they release the code, data, and trained models. What conditions and efficiency measure underlie the reported comparison between Ibis0 and the best frontier model in the sequential experiment selection task? ## Original source arXiv cs.AI · Article Original source date: Oct 29, 2026 (2026-10-29). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-10T11:34:16.028Z. First observed: 2026-11-02T05:01:23.175Z. Retrieved: 2026-11-02T10:46:17.526Z. - [Do Language Models Know What Compelling Scientific Evidence Looks Like?](https://arxiv.org/abs/2610.41015) ## Limits The available evidence is the paper's abstract; it does not give the methods or conditions needed to assess the reported 2.5-times efficiency comparison. The abstract describes three biological domains and a sequential experiment selection task, but does not establish how the reported results transfer to other scientific domains or real-world discovery. ## Open question What conditions and efficiency measure underlie the reported comparison between Ibis0 and the best frontier model in the sequential experiment selection task? ## Source-stated The authors introduce EDB-Bio, a benchmark based on real-world data from three biological domains, with episodes asking models to estimate the probability of a target hypothesis after seeing experimental results. > Each of EDB-Bio's 96k episodes presents a sequence of experimental results from published studies and asks the model to predict the probability that a target hypothesis is true. - [Supporting source](https://arxiv.org/abs/2610.41015) ## Source-stated The authors report systematic underconfidence and Brier scores comparable to a trivial base-rate forecaster for open-weight and frontier models without additional training. > Out-of-the-box, open-weight and frontier models alike demonstrate systematic underconfidence and attain Brier scores comparable to a trivial base-rate forecaster. - [Supporting source](https://arxiv.org/abs/2610.41015) ## Source-stated The authors report that post-training against strictly proper scoring rules improves calibration and discrimination, and that Ibis0 is more efficient in their sequential experiment selection task. > Post-training against strictly proper scoring rules significantly improves both calibration and discrimination. This calibration pays off downstream: in a sequential experiment selection task, our Ibis0 model is 2.5 times more efficient than the best frontier model. - [Supporting source](https://arxiv.org/abs/2610.41015) ## Source-stated The authors use Bayes factors to analyze model belief updates, agreement with expert biologists about evidence relevance, and inconsistencies and biases. > quantifying evidence weights as Bayes factors reveals how models update their beliefs over time, whether they agree with expert biologists on relevance, and where they exhibit inconsistencies and biases. - [Supporting source](https://arxiv.org/abs/2610.41015) ## Source and contributor context Igoe, Conor, Grand, Gabriel, Koch, Zane, Bou, Albert, Moss, Eli, Hinks, Michaela M., Rodriques, Samuel G., White, Andrew, Narayanan, Siddharth · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Open weights A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted. - [Definition reference](https://opensource.org/ai/open-weights) ## Frontier model One of the most capable AI models available at a given time; the set shifts as models improve. - [Definition reference](https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 9d580cff-87c7-432e-835a-039d2a0b7f84. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T11:34:16.028Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 202bf11c950fa49e01879d52c239c78b93c5ec95c07ddbf87d7ff4d848e14014. Source text hash: a6313d2ad3e2e0fedad891eb8c6e067b8282672f47a41d0809918e0234fd08aa. ### uncertain — 2026-11-10T11:34:16.028Z — anthropic/claude-opus-5.5 Publication: 9d580cff-87c7-432e-835a-039d2a0b7f84 attribution — pass: The reading attributes the benchmark, results, the Ibis0 comparison and the release to the paper's authors, using hedges such as 'They report' and 'is reported to be'. The arXiv cs.AI label and paper title match the source. > Authors:Conor Igoe, Gabriel Grand, Zane Koch, Albert Bou, Eli Moss, Michaela M. Hinks, Samuel G. Rodriques, Andrew White, Siddharth Narayanan summary — uncertain: The reading is faithful, but its central findings depend on unexplained terms: Brier scores, trivial base-rate forecaster, calibration, discrimination, strictly proper scoring rules, Bayes factors and open-weight/frontier. A general reader cannot tell that the untrained models scored about as well as a baseline that always predicts the base rate. It also omits the stated stakes, that language models increasingly drive autonomous discovery systems, and does not say Ibis0 is the authors' post-trained model. Smallest repair: add that motivation, identify Ibis0, and link the key terms to tooltips or gloss them briefly. > To what extent are language models, which are increasingly driving autonomous discovery systems, capable of such evidence deliberation? date — pass: The originalPublished value of 2026-10-29 matches the v1 submission date shown on the arXiv page. The later feed date, 2026-11-02, is correctly not used. > \[Submitted on 29 Oct 2026\] support — pass: Each claim matches the abstract, and the excerpts are verbatim. Hedged wording ('reported', 'without additional training') stays within the source. The limitations and the question correctly note what the abstract leaves unestablished. No benefits or costs are invented. > in a sequential experiment selection task, our Ibis0 model is 2.5 times more efficient than the best frontier model. media — pass: The selected media is the verified, direct link to the arXiv abstract page for this paper, which is the original source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/38ca62e34d6be8c483070150883ee83b2e13e9cfe411a7c5599d5927dec2ba74/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-do-language-models-know-what-compelling-scientific-38ca62e3.json)