Jiang, Yizhen, Guo, Haowei, Bai, Tianyi, Hu, Zengjie, Ma, Lu, Zhao, Zhengyang, Sun, Haoze, Pei, Peng, Zhang, Wentao · Nov 5, 2026
arXiv cs.AI: MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents
Markdown · JSON · Original source
The MCPHallu authors introduce a for testing how agents using the Model Context Protocol (MCP)—a system for discovering and invoking tools across servers—fail through reasoning, execution, and memory errors. They argue that existing benchmarks primarily measure task completion and, where they diagnose failures, do so after execution, while MCPHallu builds specific failure risks into tasks. Its 358 tasks run on 36 production MCP servers in isolated containers and target four subtypes: Branch Collapse, Unreachable Goal, Tool Misuse, and Context Forgetting. Across 5,012 trajectories from fourteen agents by eight vendors, the authors report an average score of 0.642 on a scale where higher is better. Scores fall to 0.458 on Unreachable Goal tasks, and 23.2% of runs pass; the authors say agents often keep calling tools or claim success instead of recognizing infeasible goals. Model rankings also vary across subtypes, so the authors argue that aggregate task success can hide distinct weaknesses. The results describe performance on this benchmark; the abstract does not establish how well they predict behavior in other settings.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 5, 2026 (2026-11-05).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-08T03:02:40.987Z. First observed: 2026-11-06T05:07:17.485Z. Retrieved: 2026-11-08T02:33:04.991Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The available evidence is the paper's abstract; it does not provide task-level results, detailed evaluation procedures, or further evidence for interpreting the reported scores. The reported results concern the 's 36 production MCP servers; the abstract does not establish whether they generalize to other environments.
Open question
Would the subtype rankings and Unreachable Goal results hold in MCP environments beyond the 36 production servers tested?
Source-stated
The authors introduce MCPHallu as a benchmark that builds reasoning, execution, and memory failure risks into tasks, rather than diagnosing failures only after execution.
MCPHallu, a benchmark for diagnosing hallucination-prone agent failures in MCP environments along three axes: reasoning, execution, and memory.
Source-stated
MCPHallu includes 358 tasks across five capability domains, run on 36 production MCP servers in isolated containers, with four targeted failure subtypes.
MCPHallu contains 358 tasks across five capability domains and runs on 36 production MCP servers in isolated containers.
Source-stated
In the authors' evaluation of fourteen agents from eight vendors across 5,012 trajectories, the average score was 0.642 on a scale where higher is better.
Current MCP agents remain vulnerable to hallucination-inducing tool environments, achieving an average score of 0.642 on a $[0,1]$ scale (higher is better).
Source-stated
The authors report lower scores on Unreachable Goal tasks and say agents often continue making tool calls or claim completion instead of recognizing infeasible goals.
the average score drops to 0.458 and 23.2\% of runs pass, indicating that agents often continue issuing tool calls or claim successful completion rather than recognizing infeasible goals.
Source-stated
The authors report that model rankings differ across failure subtypes, arguing that aggregate task success alone may miss distinct weaknesses.
a model's rank shifts by 6.9 of 13 positions on average across subtypes
Source and contributor context
Jiang, Yizhen, Guo, Haowei, Bai, Tianyi, Hu, Zengjie, Ma, Lu, Zhao, Zhengyang, Sun, Haoze, Pei, Peng, Zhang, Wentao · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Hallucination
When an AI system states something false or made up as if it were true.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: f5ec3464-d9e1-4ba2-8761-56e27407f01e.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-08T03:02:40.987Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 75c62cc85e746ba780b5cd7490a6f1dd9a98c5f4440eb71d196c17d212607d4f. Source text hash: 4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b.
Revision and source-check history
Version 1 · 2026-11-08T03:02:40.987Z
arXiv cs.AI: MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents
The MCPHallu authors introduce a for testing how agents using the Model Context Protocol (MCP)—a system for discovering and invoking tools across servers—fail through reasoning, execution, and memory errors. They argue that existing benchmarks primarily measure task completion and, where they diagnose failures, do so after execution, while MCPHallu builds specific failure risks into tasks. Its 358 tasks run on 36 production MCP servers in isolated containers and target four subtypes: Branch Collapse, Unreachable Goal, Tool Misuse, and Context Forgetting. Across 5,012 trajectories from fourteen agents by eight vendors, the authors report an average score of 0.642 on a scale where higher is better. Scores fall to 0.458 on Unreachable Goal tasks, and 23.2% of runs pass; the authors say agents often keep calling tools or claim success instead of recognizing infeasible goals. Model rankings also vary across subtypes, so the authors argue that aggregate task success can hide distinct weaknesses. The results describe performance on this benchmark; the abstract does not establish how well they predict behavior in other settings.
Original source
cs.AI · Article
Source details
Original source date: Nov 5, 2026 (2026-11-05).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-08T03:02:40.987Z. First observed: 2026-11-06T05:07:17.485Z. Retrieved: 2026-11-08T02:33:04.991Z.
Limits
The available evidence is the paper's abstract; it does not provide task-level results, detailed evaluation procedures, or further evidence for interpreting the reported scores. The reported results concern the 's 36 production MCP servers; the abstract does not establish whether they generalize to other environments.
Open question
Would the subtype rankings and Unreachable Goal results hold in MCP environments beyond the 36 production servers tested?
Source-stated
The authors introduce MCPHallu as a benchmark that builds reasoning, execution, and memory failure risks into tasks, rather than diagnosing failures only after execution.
MCPHallu, a benchmark for diagnosing hallucination-prone agent failures in MCP environments along three axes: reasoning, execution, and memory.
Source-stated
MCPHallu includes 358 tasks across five capability domains, run on 36 production MCP servers in isolated containers, with four targeted failure subtypes.
MCPHallu contains 358 tasks across five capability domains and runs on 36 production MCP servers in isolated containers.
Source-stated
In the authors' evaluation of fourteen agents from eight vendors across 5,012 trajectories, the average score was 0.642 on a scale where higher is better.
Current MCP agents remain vulnerable to hallucination-inducing tool environments, achieving an average score of 0.642 on a $[0,1]$ scale (higher is better).
Source-stated
The authors report lower scores on Unreachable Goal tasks and say agents often continue making tool calls or claim completion instead of recognizing infeasible goals.
the average score drops to 0.458 and 23.2\% of runs pass, indicating that agents often continue issuing tool calls or claim successful completion rather than recognizing infeasible goals.
Source-stated
The authors report that model rankings differ across failure subtypes, arguing that aggregate task success alone may miss distinct weaknesses.
a model's rank shifts by 6.9 of 13 positions on average across subtypes
Source and contributor context
Jiang, Yizhen, Guo, Haowei, Bai, Tianyi, Hu, Zengjie, Ma, Lu, Zhao, Zhengyang, Sun, Haoze, Pei, Peng, Zhang, Wentao · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Hallucination
When an AI system states something false or made up as if it were true.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: f5ec3464-d9e1-4ba2-8761-56e27407f01e.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-08T03:02:40.987Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 75c62cc85e746ba780b5cd7490a6f1dd9a98c5f4440eb71d196c17d212607d4f. Source text hash: 4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b.
passed · 2026-11-08T03:02:40.987Z · anthropic/claude-opus-5.5
Publication: f5ec3464-d9e1-4ba2-8761-56e27407f01e
attribution — pass: The reading attributes the benchmark, its design rationale and the results to the MCPHallu authors, with arXiv cs.AI as the publisher. Findings stay framed as the authors' reports and arguments rather than established fact.
We introduce MCPHallu, a benchmark for diagnosing hallucination-prone agent failures in MCP environments along three axes: reasoning, execution, and memory.
summary — pass: The summary leads with the actor and the development, and explains MCP in plain terms. It conveys the core finding: agents keep calling tools or claim success on infeasible goals, and rankings shift by subtype. The abstract does not define Branch Collapse, Tool Misuse, Context Forgetting or 'trajectories', so leaving them unexplained is acceptable; tooltips could help. Calling the rankings 'vary' slightly softens the source's 'diverge sharply' but does not change its meaning. The question follows from the stated generalization gap.
Subtype rankings diverge sharply: a model's rank shifts by 6.9 of 13 positions on average across subtypes
date — pass: The originalPublished date of 2026-11-05 matches the arXiv v1 submission date. The feed date of 2026-11-06 is later and is correctly not used.
[Submitted on 5 Nov 2026]
support — pass: The counts (358 tasks, 36 servers, fourteen agents, eight vendors, 5,012 trajectories) and the scores 0.642, 0.458 and 23.2% match the abstract. The point that aggregate success can hide weaknesses is a reasonable restatement of the subtype-level design rationale and the reported rank divergence. The limitations about generalization are appropriately conditional and invent no benefits or costs.
This design enables subtype-level analysis rather than aggregate task success alone.
media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most direct way to reach the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference