Escape Meridian

Galeone, Cosimo, Ettorre, Anna, Mariani, Ginevra, Park, Minsu, Ettorre, Giuseppe, Ligorio, Daniele · Nov 13, 2026

Can You Trust Your Finance Agent? Drift, Diagnosis and Auditable Intervention on One Shipped Financial Research Agent

Markdown · JSON · Original source

The paper’s authors test the reference agent shipped with the Vals Finance Agent v2, without tuning it, asking whether its score is stable, failures can be diagnosed, and changes can be trusted. Seven identical runs on 18 questions passed between 105 and 123 of 149 rubric criteria, while the bill varied by up to 17%. In 20 failed trajectories, the authors assigned responsibility to the layer whose owner would remedy each failure: 13 to the agent’s instructions, 3 to its tools, 4 to the model, and none to its data. Four blind readers agreed with the authors’ analyzer in 94% of cases, close to their 93% agreement with one another. An optimization loop kept a targeted change only when it cleared a fixed threshold. Under a cost objective, the final configuration cost 0.34x the baseline on held-out questions, with accuracy within noise. Under an accuracy objective, it gained 14 points on development questions and 1.5 on held-out questions, which the authors judge not to be a real gain against measured run-to-run noise. They emphasize that this is a case study of one agent and nine test questions, not a general result.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

arXiv cs.AIArticle · arxiv.orgOpen original source

Original source

cs.AI · Article

Source details

Original source date: Nov 13, 2026 (2026-11-13).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-16T07:16:19.524Z. First observed: 2026-11-16T05:06:36.888Z. Retrieved: 2026-11-16T07:03:20.313Z.

Referenced by

Recorded citations in public readings. A reference alone does not establish agreement or influence.

No public references recorded here yet.

Reference data (JSON)

Limits

The authors describe this as a case study of one agent and nine test questions, so the evidence does not establish that the results generalize to other agents or tests. The evidence is the abstract; detailed methods and results beyond what it reports are not available here.

Open question

Would the same pattern of run-to-run variation, failure ownership, and optimization results appear with other agents or test questions?

Source-stated

The authors tested the untuned reference agent from the Vals Finance Agent Benchmark v2 across repeated runs and found variation in both rubric performance and cost.

Seven identical runs on 18 questions pass between 105 and 123 of 149 rubric criteria, and the bill varies by up to 17%.

Source-stated

The authors classified failures by the layer whose owner would remedy them; most were assigned to the agent’s instructions, with none assigned to its data.

13 belong to the agent's instructions, 3 to its tools, 4 to the model, none to its data.

Source-stated

Blind readers given traces agreed with the authors’ LLM analyzer at a rate similar to the readers’ agreement with one another.

name the same fault as our LLM analyzer in 94% of cases, the rate at which they agree with one another (93%).

Source-stated

The authors report that a cost-focused optimization reduced cost on held-out questions while accuracy stayed within noise; an accuracy-focused optimization did not yield a meaningful held-out gain under their prewritten rule.

Under a cost objective its final configuration costs 0.34x the baseline on the held-out questions (95% CI [0.22, 0.58]) at accuracy within noise.

Source-stated

The authors caution that their findings are limited to one agent and nine test questions.

This is a case study of one agent and nine test questions, not a general result.

Source and contributor context

Galeone, Cosimo, Ettorre, Anna, Mariani, Ginevra, Park, Minsu, Ettorre, Giuseppe, Ligorio, Daniele · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.

Source media

No suitable media was found by the supported page extraction.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

Large language model

A language model with many learned numerical settings, used to process and generate language.

arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

Editorial provenance

Version 1. Publication ID: f237075d-4904-45d5-84ce-31ee7e29ce0b.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-16T07:16:19.524Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material_development.

Observation: fad274f0488444fc0388812c94843bb6db10a7bd10d05408f392e406c97ccc11. Source text hash: 59f25a1983404a763b42905337355611d1af35c9d9a7705186c4a4cf6d6d409d.

Revision and source-check history

Version 1 · 2026-11-16T07:16:19.524Z

Can You Trust Your Finance Agent? Drift, Diagnosis and Auditable Intervention on One Shipped Financial Research Agent

The paper’s authors test the reference agent shipped with the Vals Finance Agent v2, without tuning it, asking whether its score is stable, failures can be diagnosed, and changes can be trusted. Seven identical runs on 18 questions passed between 105 and 123 of 149 rubric criteria, while the bill varied by up to 17%. In 20 failed trajectories, the authors assigned responsibility to the layer whose owner would remedy each failure: 13 to the agent’s instructions, 3 to its tools, 4 to the model, and none to its data. Four blind readers agreed with the authors’ analyzer in 94% of cases, close to their 93% agreement with one another. An optimization loop kept a targeted change only when it cleared a fixed threshold. Under a cost objective, the final configuration cost 0.34x the baseline on held-out questions, with accuracy within noise. Under an accuracy objective, it gained 14 points on development questions and 1.5 on held-out questions, which the authors judge not to be a real gain against measured run-to-run noise. They emphasize that this is a case study of one agent and nine test questions, not a general result.

Original source

cs.AI · Article

Source details

Original source date: Nov 13, 2026 (2026-11-13).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-16T07:16:19.524Z. First observed: 2026-11-16T05:06:36.888Z. Retrieved: 2026-11-16T07:03:20.313Z.

Limits

The authors describe this as a case study of one agent and nine test questions, so the evidence does not establish that the results generalize to other agents or tests. The evidence is the abstract; detailed methods and results beyond what it reports are not available here.

Open question

Would the same pattern of run-to-run variation, failure ownership, and optimization results appear with other agents or test questions?

Source-stated

The authors tested the untuned reference agent from the Vals Finance Agent Benchmark v2 across repeated runs and found variation in both rubric performance and cost.

Seven identical runs on 18 questions pass between 105 and 123 of 149 rubric criteria, and the bill varies by up to 17%.

Source-stated

The authors classified failures by the layer whose owner would remedy them; most were assigned to the agent’s instructions, with none assigned to its data.

13 belong to the agent's instructions, 3 to its tools, 4 to the model, none to its data.

Source-stated

Blind readers given traces agreed with the authors’ LLM analyzer at a rate similar to the readers’ agreement with one another.

name the same fault as our LLM analyzer in 94% of cases, the rate at which they agree with one another (93%).

Source-stated

The authors report that a cost-focused optimization reduced cost on held-out questions while accuracy stayed within noise; an accuracy-focused optimization did not yield a meaningful held-out gain under their prewritten rule.

Under a cost objective its final configuration costs 0.34x the baseline on the held-out questions (95% CI [0.22, 0.58]) at accuracy within noise.

Source-stated

The authors caution that their findings are limited to one agent and nine test questions.

This is a case study of one agent and nine test questions, not a general result.

Source and contributor context

Galeone, Cosimo, Ettorre, Anna, Mariani, Ginevra, Park, Minsu, Ettorre, Giuseppe, Ligorio, Daniele · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.

Source media

No suitable media was found by the supported page extraction.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

Large language model

A language model with many learned numerical settings, used to process and generate language.

arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

Editorial provenance

Version 1. Publication ID: f237075d-4904-45d5-84ce-31ee7e29ce0b.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-16T07:16:19.524Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material_development.

Observation: fad274f0488444fc0388812c94843bb6db10a7bd10d05408f392e406c97ccc11. Source text hash: 59f25a1983404a763b42905337355611d1af35c9d9a7705186c4a4cf6d6d409d.

uncertain · 2026-11-16T07:16:19.524Z · anthropic/claude-opus-5.5

Publication: f237075d-4904-45d5-84ce-31ee7e29ce0b

attribution — pass: The reading consistently attributes the questions, results, and judgments to the paper's authors. It does not present the claims as established facts or as Escape Meridian's opinion.

Can you trust a finance agent you did not build?

summary — uncertain: Mostly faithful, but two problems affect meaning. First, the summary drops 'from three model families', so readers will likely assume the four blind readers were humans; they appear to be AI models. Smallest repair: 'four blind model-based readers from three model families'. Second, central jargon is unexplained for a general reader: finance agent, reference agent, Vals benchmark, rubric criteria, trajectories, LLM analyzer, and held-out versus development questions. Tooltips for these terms would close the gap. The question is a fair, open issue that follows from the source.

Four blind readers from three model families, given only the traces, name the same fault as our LLM analyzer in 94% of cases

date — pass: The originalPublished date of 2026-11-13 matches the arXiv v1 submission date. The later feed date is not used as the publication date.

[Submitted on 13 Nov 2026]

support — pass: Each claim's excerpt is verbatim and matches the abstract: score and cost variation, fault ownership counts, 94% versus 93% agreement, the 0.34x cost result, and no real held-out accuracy gain under the prewritten rule. The case-study limitation is also stated in the source. No unsupported benefits or costs are implied.

Judged against the measured run-to-run noise, under a rule written before the test was opened, the first is a real saving and the second is not.

media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most direct way for readers to access the source.

Challenge or correct this reading

Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.