Cao, Yupeng, Peng, Xueqing, Qian, Lingfei, Li, Haohang, Yu, Yangyang, Xu, Anke, Cao, Wenbo, Wang, Yan, Jiang, Yuechen, Wang, Yuyan, Xie, Zhuohan, Chen, Yankai, Li, Zehan, Huang, Jimin, Liu, Xue, Nie, Jian-Yun, Ananiadou, Sophia · Nov 5, 2026
arXiv cs.AI paper proposes an open-world test for financial agents
Markdown · JSON · Original source
Yupeng Cao and coauthors introduce FINCOSMOS, an open-world evaluation paradigm for agents doing financial work. Unlike that fix the task, data, or timeline, FINCOSMOS gives a simulated user a hidden strategy and shares it only when the agent asks. The agent must acquire data from the Web, build its own backtest under a data cutoff, and deliver verifiable results. Programs check point-in- correctness and recompute reported metrics; the delivered strategy is checked against the hidden specification. A run counts as accepted only if the user is satisfied and none of 15 veto checks fails. In 150 sessions involving four coding agents and a model without tools, no session was accepted. Every session signed off by the simulated user failed a program check, and agents raised only 54–64% of details they had not been told. Human experts attributed 88.33% of failures to the agents. The authors also report that, with FINCOSMOS feedback, an agent with persistent memory learned all ten tasks. As an Escape Meridian interpretation, these results make the paper’s concern concrete: a financial report can look convincing while relying on an unasked-for strategy, unverified data, or a backtest that uses the future. The evidence comes from this evaluation; it does not establish how agents perform in financial work outside it.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 5, 2026 (2026-11-05).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-08T19:24:11.077Z. First observed: 2026-11-06T05:12:38.913Z. Retrieved: 2026-11-08T19:17:06.783Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The reported results are from 150 sessions involving four coding agents and a model without tools; the evidence does not establish performance in financial work outside this evaluation. The abstract reports that an agent with persistent memory learned all ten tasks with FINCOSMOS feedback, but does not describe those tasks or the learning procedure in detail.
Open question
What did the agent with persistent memory do to learn all ten tasks with FINCOSMOS feedback?
Source-stated
The authors propose FINCOSMOS, an open-world evaluation paradigm for agents doing financial work, in which agents must obtain Web data and build a backtest under a data cutoff.
We introduce FINCOSMOS, an open-world evaluation paradigm for agents doing financial work. A simulated user holds a hidden strategy sampled from a compositional grammar and reveals it only when asked.
Source-stated
FINCOSMOS checks reported financial work programmatically and accepts a run only when the simulated user is satisfied and none of 15 veto checks fails.
Programs verify point-in-time correctness and recompute every reported metric, the delivered strategy is checked against the hidden specification, and a run is accepted only when the user is satisfied and none of 15 veto checks fails.
Source-stated
In the reported 150 sessions, no session was accepted; every simulated-user sign-off failed a program check, and agents raised only 54–64% of details they were not told.
Across four coding agents and a model without tools in 150 sessions, no session is accepted: every session that the simulated user signed off fails a program check, and agents raise only 54--64\% of the details they were not told.
Source-stated
The authors report that human experts attributed 88.33% of failures to agents, and that an agent with persistent memory learned all ten tasks with FINCOSMOS feedback.
Human experts attribute 88.33\% of failures to the agents. With FINCOSMOS feedback, an agent with persistent memory learned all ten tasks.
Escape Meridian interpretation
The evaluation illustrates the authors’ concern that financial work can yield convincing reports despite strategy, data, or timing errors.
a strategy the user never asked for, unverified data, or a backtest that uses the future yields a report as convincing as a correct one.
Source and contributor context
Cao, Yupeng, Peng, Xueqing, Qian, Lingfei, Li, Haohang, Yu, Yangyang, Xu, Anke, Cao, Wenbo, Wang, Yan, Jiang, Yuechen, Wang, Yuyan, Xie, Zhuohan, Chen, Yankai, Li, Zehan, Huang, Jimin, Liu, Xue, Nie, Jian-Yun, Ananiadou, Sophia · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 3800a478-5f84-4ab3-89ff-e0863b9acba9.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-08T19:24:11.077Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: ed999760e82736ca485a173e9f04bb01f28531153cc5ea786aafeb7c15a4ad12. Source text hash: c3afa81a89f4895026b220d790c3204b1fb137d2b9f7a6c7dc46cf6a6c5d3bf3.
Revision and source-check history
Version 1 · 2026-11-08T19:24:11.077Z
arXiv cs.AI paper proposes an open-world test for financial agents
Yupeng Cao and coauthors introduce FINCOSMOS, an open-world evaluation paradigm for agents doing financial work. Unlike that fix the task, data, or timeline, FINCOSMOS gives a simulated user a hidden strategy and shares it only when the agent asks. The agent must acquire data from the Web, build its own backtest under a data cutoff, and deliver verifiable results. Programs check point-in- correctness and recompute reported metrics; the delivered strategy is checked against the hidden specification. A run counts as accepted only if the user is satisfied and none of 15 veto checks fails. In 150 sessions involving four coding agents and a model without tools, no session was accepted. Every session signed off by the simulated user failed a program check, and agents raised only 54–64% of details they had not been told. Human experts attributed 88.33% of failures to the agents. The authors also report that, with FINCOSMOS feedback, an agent with persistent memory learned all ten tasks. As an Escape Meridian interpretation, these results make the paper’s concern concrete: a financial report can look convincing while relying on an unasked-for strategy, unverified data, or a backtest that uses the future. The evidence comes from this evaluation; it does not establish how agents perform in financial work outside it.
Original source
cs.AI · Article
Source details
Original source date: Nov 5, 2026 (2026-11-05).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-08T19:24:11.077Z. First observed: 2026-11-06T05:12:38.913Z. Retrieved: 2026-11-08T19:17:06.783Z.
Limits
The reported results are from 150 sessions involving four coding agents and a model without tools; the evidence does not establish performance in financial work outside this evaluation. The abstract reports that an agent with persistent memory learned all ten tasks with FINCOSMOS feedback, but does not describe those tasks or the learning procedure in detail.
Open question
What did the agent with persistent memory do to learn all ten tasks with FINCOSMOS feedback?
Source-stated
The authors propose FINCOSMOS, an open-world evaluation paradigm for agents doing financial work, in which agents must obtain Web data and build a backtest under a data cutoff.
We introduce FINCOSMOS, an open-world evaluation paradigm for agents doing financial work. A simulated user holds a hidden strategy sampled from a compositional grammar and reveals it only when asked.
Source-stated
FINCOSMOS checks reported financial work programmatically and accepts a run only when the simulated user is satisfied and none of 15 veto checks fails.
Programs verify point-in-time correctness and recompute every reported metric, the delivered strategy is checked against the hidden specification, and a run is accepted only when the user is satisfied and none of 15 veto checks fails.
Source-stated
In the reported 150 sessions, no session was accepted; every simulated-user sign-off failed a program check, and agents raised only 54–64% of details they were not told.
Across four coding agents and a model without tools in 150 sessions, no session is accepted: every session that the simulated user signed off fails a program check, and agents raise only 54--64\% of the details they were not told.
Source-stated
The authors report that human experts attributed 88.33% of failures to agents, and that an agent with persistent memory learned all ten tasks with FINCOSMOS feedback.
Human experts attribute 88.33\% of failures to the agents. With FINCOSMOS feedback, an agent with persistent memory learned all ten tasks.
Escape Meridian interpretation
The evaluation illustrates the authors’ concern that financial work can yield convincing reports despite strategy, data, or timing errors.
a strategy the user never asked for, unverified data, or a backtest that uses the future yields a report as convincing as a correct one.
Source and contributor context
Cao, Yupeng, Peng, Xueqing, Qian, Lingfei, Li, Haohang, Yu, Yangyang, Xu, Anke, Cao, Wenbo, Wang, Yan, Jiang, Yuechen, Wang, Yuyan, Xie, Zhuohan, Chen, Yankai, Li, Zehan, Huang, Jimin, Liu, Xue, Nie, Jian-Yun, Ananiadou, Sophia · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 3800a478-5f84-4ab3-89ff-e0863b9acba9.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-08T19:24:11.077Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: ed999760e82736ca485a173e9f04bb01f28531153cc5ea786aafeb7c15a4ad12. Source text hash: c3afa81a89f4895026b220d790c3204b1fb137d2b9f7a6c7dc46cf6a6c5d3bf3.
uncertain · 2026-11-08T19:24:11.077Z · anthropic/claude-opus-5.5
Publication: 3800a478-5f84-4ab3-89ff-e0863b9acba9
attribution — pass: The draft attributes FINCOSMOS to Yupeng Cao and coauthors in an arXiv cs.AI paper. This matches the author list. Results are framed as reported by the authors, and the Escape Meridian interpretation is labeled as such.
Authors:Yupeng Cao, Xueqing Peng, Lingfei Qian, Haohang Li
summary — uncertain: The summary is faithful, and the title and question fit the source. However, central finance terms are never explained in plain language: backtest (testing a strategy on past data), data cutoff, point-in-time correctness and veto checks. Without them, a non-specialist cannot see why a backtest that uses the future is a silent failure. Smallest repair: add a brief plain-language gloss of backtest and data cutoff, or term-library tooltips. The draft also omits that the agent receives no data, which is minor but explains why it must use the Web.
The agent receives no data, so it must acquire data from the Web, build its own backtest under a data cutoff, and deliver verifiable results.
date — pass: The originalPublished date of 2026-11-05 matches the v1 submission date. It is not the later feed date of 2026-11-06.
[Submitted on 5 Nov 2026]
support — pass: All numeric claims match the abstract: 150 sessions, four coding agents plus a model without tools, zero accepted sessions, 54–64%, 88.33%, ten tasks and 15 veto checks. The interpretation is labeled, tied to the authors' stated concern, and bounded by a limitation against generalizing beyond the evaluation. No benefits or costs are invented.
Across four coding agents and a model without tools in 150 sessions, no session is accepted: every session that the simulated user signed off fails a program check
media — pass: The selected media is the verified, direct arXiv abstract page for this paper, which is the original source. It is relevant and easy to access.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv