# Benchmarking Forecasting Agents on Past Events [Canonical reading](https://escapemeridian.com/readings/benchmarking-forecasting-agents-on-past-events-1f844d93) Yuxuan Wang, Yishu Wang, and Jiaqi Deng introduce FABench, a backtesting environment for evaluating forecasting agents on past events. Forecasting tests whether agents can combine incomplete evidence, infer causes, and make calibrated probabilistic judgments. The authors argue that existing approaches have a timing trade-off: live benchmarks must wait for events to resolve, while fixed-corpus evaluations use documents collected in advance. FABench generates questions for past forecast dates and lets agents research them using an offline archive and live web search screened to prevent temporal leakage. The authors report that retrieval improved the Brier score—a measure of probabilistic forecast accuracy—for all eight evaluated models in every retrieval setting. On questions linked to Polymarket, the leading agents reached market consensus, with the best three scoring slightly above it. These results describe the benchmark evaluations, not forecasting performance in every real-world setting. The paper also reports recurring failures in how agents weigh and apply evidence, but the abstract does not describe those failures. The authors release the code, data, and archive environment for further research. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Oct 29, 2026 (2026-10-29). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-10T11:39:30.218Z. First observed: 2026-11-02T05:01:07.281Z. Retrieved: 2026-11-02T05:01:13.741Z. - [Benchmarking Forecasting Agents on Yesterday's Tomorrows](https://arxiv.org/abs/2610.40940) ## Limits The supplied evidence is the arXiv abstract, not the full paper; it does not detail the recurring evidence-weighing failures or provide the evaluation methods and results needed to assess them. The reported results concern the benchmark evaluations described in the abstract; the supplied evidence does not establish how well the agents forecast outside those evaluations. ## Open question What recurring failures in how agents weigh and apply evidence did the research traces reveal? ## Source-stated The authors introduce FABench, a backtesting environment that creates forecasting questions for past dates and supports agent research through an offline archive and screened live web search. > An automated pipeline discovers events, generates questions, and resolves outcomes for any past forecast date - [Supporting source](https://arxiv.org/abs/2610.40940) ## Source-stated The authors report that retrieval improved the Brier score of all eight evaluated models in every retrieval setting. > retrieval improves the Brier score of all eight in every retrieval setting - [Supporting source](https://arxiv.org/abs/2610.40940) ## Source-stated For the 54 questions linked to Polymarket, the authors say the leading agents reached market consensus, with the best three scoring slightly above it. > lifts the leading agents to the market consensus on the 54 questions linked to Polymarket, with the best three scoring slightly above it - [Supporting source](https://arxiv.org/abs/2610.40940) ## Source-stated The authors report recurring failures in how agents weigh and apply evidence, but the abstract does not specify what those failures are. > Research traces reveal recurring failures in how agents weigh and apply evidence - [Supporting source](https://arxiv.org/abs/2610.40940) ## Source and contributor context Wang, Yuxuan, Wang, Yishu, Deng, Jiaqi · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 79bba150-a663-422f-806c-d393dafa5d44. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T11:39:30.218Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 0ac666c159b8f0eb0a7e073b9d43e937e4bb08c9b61052f5df10f4a970a3b764. Source text hash: 9af0dc2c7a6e9299fca43e1137d10851930352c2c8a962fae6793c482bac8397. ## Revision and source-check history ### Version 1 — 2026-11-10T11:39:30.218Z Yuxuan Wang, Yishu Wang, and Jiaqi Deng introduce FABench, a backtesting environment for evaluating forecasting agents on past events. Forecasting tests whether agents can combine incomplete evidence, infer causes, and make calibrated probabilistic judgments. The authors argue that existing approaches have a timing trade-off: live benchmarks must wait for events to resolve, while fixed-corpus evaluations use documents collected in advance. FABench generates questions for past forecast dates and lets agents research them using an offline archive and live web search screened to prevent temporal leakage. The authors report that retrieval improved the Brier score—a measure of probabilistic forecast accuracy—for all eight evaluated models in every retrieval setting. On questions linked to Polymarket, the leading agents reached market consensus, with the best three scoring slightly above it. These results describe the benchmark evaluations, not forecasting performance in every real-world setting. The paper also reports recurring failures in how agents weigh and apply evidence, but the abstract does not describe those failures. The authors release the code, data, and archive environment for further research. ## Original source arXiv cs.AI · Article Original source date: Oct 29, 2026 (2026-10-29). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-10T11:39:30.218Z. First observed: 2026-11-02T05:01:07.281Z. Retrieved: 2026-11-02T05:01:13.741Z. - [Benchmarking Forecasting Agents on Yesterday's Tomorrows](https://arxiv.org/abs/2610.40940) ## Limits The supplied evidence is the arXiv abstract, not the full paper; it does not detail the recurring evidence-weighing failures or provide the evaluation methods and results needed to assess them. The reported results concern the benchmark evaluations described in the abstract; the supplied evidence does not establish how well the agents forecast outside those evaluations. ## Open question What recurring failures in how agents weigh and apply evidence did the research traces reveal? ## Source-stated The authors introduce FABench, a backtesting environment that creates forecasting questions for past dates and supports agent research through an offline archive and screened live web search. > An automated pipeline discovers events, generates questions, and resolves outcomes for any past forecast date - [Supporting source](https://arxiv.org/abs/2610.40940) ## Source-stated The authors report that retrieval improved the Brier score of all eight evaluated models in every retrieval setting. > retrieval improves the Brier score of all eight in every retrieval setting - [Supporting source](https://arxiv.org/abs/2610.40940) ## Source-stated For the 54 questions linked to Polymarket, the authors say the leading agents reached market consensus, with the best three scoring slightly above it. > lifts the leading agents to the market consensus on the 54 questions linked to Polymarket, with the best three scoring slightly above it - [Supporting source](https://arxiv.org/abs/2610.40940) ## Source-stated The authors report recurring failures in how agents weigh and apply evidence, but the abstract does not specify what those failures are. > Research traces reveal recurring failures in how agents weigh and apply evidence - [Supporting source](https://arxiv.org/abs/2610.40940) ## Source and contributor context Wang, Yuxuan, Wang, Yishu, Deng, Jiaqi · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 79bba150-a663-422f-806c-d393dafa5d44. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T11:39:30.218Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 0ac666c159b8f0eb0a7e073b9d43e937e4bb08c9b61052f5df10f4a970a3b764. Source text hash: 9af0dc2c7a6e9299fca43e1137d10851930352c2c8a962fae6793c482bac8397. ### uncertain — 2026-11-10T11:39:30.218Z — anthropic/claude-opus-5.5 Publication: 79bba150-a663-422f-806c-d393dafa5d44 attribution — pass: The reading correctly credits Yuxuan Wang, Yishu Wang and Jiaqi Deng, as listed on the arXiv page. It keeps the results attributed with phrases such as 'the authors report' and 'the authors say'. > Authors:Yuxuan Wang, Yishu Wang, Jiaqi Deng summary — uncertain: The summary is faithful, but its central idea is hard for non-specialists to follow. It does not explain 'temporal leakage', meaning that agents could see information published after the forecast date. That risk is why testing on past events is tricky. It also omits that the benchmark can be renewed to stay ahead of model training cutoffs, so models cannot simply remember the outcomes. 'Polymarket' and 'market consensus' are also unexplained (the wisdom-of-crowds price on a prediction market). Smallest repair: add one plain sentence on leakage and renewal, and tooltip suggestions for temporal leakage, backtesting and Polymarket. > so the benchmark can be renewed to stay ahead of model training cutoffs date — pass: evidence.originalPublished is 2026-10-29. This matches the v1 arXiv submission date, not the later feed date of 2026-11-02. > \[Submitted on 29 Oct 2026\] support — pass: Each claim matches the abstract: retrieval improved the Brier score in all eight cases, leading agents reached market consensus on 54 Polymarket-linked questions, and recurring evidence-weighing failures were found. The reading adds a caveat against treating benchmark results as real-world reliability, which is appropriate. The question is a genuine gap that the abstract leaves open, and the reading implies no unsupported benefits or costs. > retrieval improves the Brier score of all eight in every retrieval setting media — pass: The selected media is the verified direct link to the original arXiv abstract page. It is the most relevant way for readers to reach the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/1f844d93840905a4797569e8256ba05d292bf98e799b871fb85a44d8fbb10f04/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/benchmarking-forecasting-agents-on-past-events-1f844d93.json)