Tambi, Ritiz, Shayegan, Sahar, Meyer, Joshua, Kim, Sun, Khan, Ali, Shih, Victor, Jamei, Mehdi, Partovi, Andi · Oct 28, 2026
An arXiv paper tests whether AI agents can search scanned financial archives
Markdown · JSON · Original source
The authors of TreasuryQA Bench introduce a for a practical task: finding figures in scanned financial records. They argue that existing benchmarks leave out important steps, either giving agents the relevant documents or letting them search text already extracted from documents. Their benchmark combines both steps. Agents search 1,435 scanned U.S. Treasury publications through Dropbox's Model Context Protocol (MCP) server, read relevant pages using OCR or their own tools, and return one exact . The authors test ten agents; the best solves 64.8% of attempts and the worst 16.7%, which they say leaves the benchmark far from saturated. The results also point to a specific weakness: every agent performs worse on the blurriest scans, and the most common cause of failure is misreading a digit. Escape Meridian interpretation: for organizations hoping to use agents on scanned records, searching for a document is not enough; reading the scan accurately also matters. The benchmark measures performance on its tasks, not whether agents can reliably handle financial records in real-world use. An open question is whether improving scan quality or OCR would reduce digit errors on this benchmark.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 28, 2026 (2026-10-28).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-13T03:32:40.338Z. First observed: 2026-10-30T06:40:37.664Z. Retrieved: 2026-10-31T11:02:48.633Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The evidence is an abstract; it does not provide details needed to assess the full experimental design or results. The reported results do not establish how agents perform on real-world financial archives. The abstract does not say whether improving scan quality or OCR would reduce errors.
Open question
Would improving scan quality or OCR reduce digit misreads on this ?
Source-stated
The authors introduce a benchmark requiring agents both to search for relevant scanned financial records and to read them.
We introduce TreasuryQA Bench, which requires both: 70 questions over 1,435 scanned U.S. Treasury publications
Source-stated
Agents search through Dropbox's MCP server, read needed pages with OCR or their own tools, and return one exact figure.
the agent searches for the relevant file through Dropbox's MCP server, reads the pages it needs with Reducto's hosted OCR server or its own tools, and replies with one exact figure.
Source-stated
Across ten agents, the best solved 64.8% of attempts and the worst 16.7%.
The best, Claude Opus 5.5 in Claude Code, solves 64.8% of attempts and the worst, GPT-5.6 Terra in Codex, solves 16.7%
Source-stated
All tested agents performed worse on the blurriest scans, and digit misreads were the most common cause of failure.
every agent solves fewer tasks on the blurriest scans. Most failed attempts reply with a wrong number rather than no number, and the most common cause is a digit misread from the scan.
Escape Meridian interpretation
The results suggest that reading scan images accurately is a distinct challenge from finding the right document.
In some, the relevant documents are handed to the agent, so it never has to search. In others, the agent searches through documents with pre-extracted text, so it never has to read scanned images.
Source and contributor context
Tambi, Ritiz, Shayegan, Sahar, Meyer, Joshua, Kim, Sun, Khan, Ali, Shih, Victor, Jamei, Mehdi, Partovi, Andi · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
AI agent
An AI system that chooses and carries out actions toward a goal, often using software tools.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Figure
Figure develops humanoid robots and publishes its own tests of how they perform physical tasks.
Editorial provenance
Version 1. Publication ID: 320722dc-69e2-4add-84bf-79171c7cf39f.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-13T03:32:40.338Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: a0e1f03e01d4b6deb509c82367cb6b0c4b62c59fa5cf2542ef8765ee8b10bf5f. Source text hash: a9eb2da725d0b7f8e0e7e912dca154378a9fe90f0a2b70e5dfdedb61aafb639b.
Revision and source-check history
Version 1 · 2026-11-13T03:32:40.338Z
An arXiv paper tests whether AI agents can search scanned financial archives
The authors of TreasuryQA Bench introduce a for a practical task: finding figures in scanned financial records. They argue that existing benchmarks leave out important steps, either giving agents the relevant documents or letting them search text already extracted from documents. Their benchmark combines both steps. Agents search 1,435 scanned U.S. Treasury publications through Dropbox's Model Context Protocol (MCP) server, read relevant pages using OCR or their own tools, and return one exact . The authors test ten agents; the best solves 64.8% of attempts and the worst 16.7%, which they say leaves the benchmark far from saturated. The results also point to a specific weakness: every agent performs worse on the blurriest scans, and the most common cause of failure is misreading a digit. Escape Meridian interpretation: for organizations hoping to use agents on scanned records, searching for a document is not enough; reading the scan accurately also matters. The benchmark measures performance on its tasks, not whether agents can reliably handle financial records in real-world use. An open question is whether improving scan quality or OCR would reduce digit errors on this benchmark.
Original source
cs.AI · Article
Source details
Original source date: Oct 28, 2026 (2026-10-28).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-13T03:32:40.338Z. First observed: 2026-10-30T06:40:37.664Z. Retrieved: 2026-10-31T11:02:48.633Z.
Limits
The evidence is an abstract; it does not provide details needed to assess the full experimental design or results. The reported results do not establish how agents perform on real-world financial archives. The abstract does not say whether improving scan quality or OCR would reduce errors.
Open question
Would improving scan quality or OCR reduce digit misreads on this ?
Source-stated
The authors introduce a benchmark requiring agents both to search for relevant scanned financial records and to read them.
We introduce TreasuryQA Bench, which requires both: 70 questions over 1,435 scanned U.S. Treasury publications
Source-stated
Agents search through Dropbox's MCP server, read needed pages with OCR or their own tools, and return one exact figure.
the agent searches for the relevant file through Dropbox's MCP server, reads the pages it needs with Reducto's hosted OCR server or its own tools, and replies with one exact figure.
Source-stated
Across ten agents, the best solved 64.8% of attempts and the worst 16.7%.
The best, Claude Opus 5.5 in Claude Code, solves 64.8% of attempts and the worst, GPT-5.6 Terra in Codex, solves 16.7%
Source-stated
All tested agents performed worse on the blurriest scans, and digit misreads were the most common cause of failure.
every agent solves fewer tasks on the blurriest scans. Most failed attempts reply with a wrong number rather than no number, and the most common cause is a digit misread from the scan.
Escape Meridian interpretation
The results suggest that reading scan images accurately is a distinct challenge from finding the right document.
In some, the relevant documents are handed to the agent, so it never has to search. In others, the agent searches through documents with pre-extracted text, so it never has to read scanned images.
Source and contributor context
Tambi, Ritiz, Shayegan, Sahar, Meyer, Joshua, Kim, Sun, Khan, Ali, Shih, Victor, Jamei, Mehdi, Partovi, Andi · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
AI agent
An AI system that chooses and carries out actions toward a goal, often using software tools.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Figure
Figure develops humanoid robots and publishes its own tests of how they perform physical tasks.
Editorial provenance
Version 1. Publication ID: 320722dc-69e2-4add-84bf-79171c7cf39f.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-13T03:32:40.338Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: a0e1f03e01d4b6deb509c82367cb6b0c4b62c59fa5cf2542ef8765ee8b10bf5f. Source text hash: a9eb2da725d0b7f8e0e7e912dca154378a9fe90f0a2b70e5dfdedb61aafb639b.
passed · 2026-11-13T03:32:40.338Z · anthropic/claude-opus-5.5
Publication: 320722dc-69e2-4add-84bf-79171c7cf39f
attribution — pass: The reading attributes the benchmark, its framing of existing benchmarks, and the results to the paper's authors. The 'far from saturated' judgment is presented as their claim, and the Escape Meridian inference is labeled.
Authors:Ritiz Tambi, Sahar Shayegan, Joshua Meyer, Sun Kim, Ali Khan, Victor Shih, Mehdi Jamei, Andi Partovi
summary — pass: The reading accurately conveys the gap in existing benchmarks, the task design, the score range, the scan-quality effect and digit misreads. A general reader can follow it, and MCP and OCR can be explained by term tooltips. One fidelity point is omitted: failed attempts mostly gave a wrong number rather than no number. A one-clause addition would restore that stake. The question follows from the evidence and is not leading.
Most failed attempts reply with a wrong number rather than no number, and the most common cause is a digit misread from the scan.
date — pass: originalPublished is 2026-10-28, which matches the v1 submission date on arXiv. The later feed date is not used.
[Submitted on 28 Oct 2026]
support — pass: The source-stated claims match the abstract verbatim. The labeled interpretation, that accurate reading matters beyond finding a document, is supported by the digit-misread and blur findings and is appropriately limited. The reading notes that benchmark scores do not establish real-world reliability, and the limitations acknowledge that the evidence is only the abstract.
every agent solves fewer tasks on the blurriest scans
media — pass: The selected media is the verified direct link to the original arXiv abstract page. It matches the source URL and is the most direct way to access the paper.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceFigure develops humanoid robots and publishes its own tests of how they perform physical tasks.
Explore FigurearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceFigure develops humanoid robots and publishes its own tests of how they perform physical tasks.
Explore FigurearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference