{"id":"aaa37d04ead994ddf52270426680b3a707742939647a34207ccfd66d8c320412","title":"arXiv cs.AI: MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents","argument":"The MCPHallu authors introduce a benchmark for testing how agents using the Model Context Protocol (MCP)—a system for discovering and invoking tools across servers—fail through reasoning, execution, and memory errors. They argue that existing benchmarks primarily measure task completion and, where they diagnose failures, do so after execution, while MCPHallu builds specific failure risks into tasks. Its 358 tasks run on 36 production MCP servers in isolated containers and target four subtypes: Branch Collapse, Unreachable Goal, Tool Misuse, and Context Forgetting.\n\nAcross 5,012 trajectories from fourteen agents by eight vendors, the authors report an average score of 0.642 on a scale where higher is better. Scores fall to 0.458 on Unreachable Goal tasks, and 23.2% of runs pass; the authors say agents often keep calling tools or claim success instead of recognizing infeasible goals. Model rankings also vary across subtypes, so the authors argue that aggregate task success can hide distinct weaknesses. The results describe performance on this benchmark; the abstract does not establish how well they predict behavior in other settings.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The available evidence is the paper's abstract; it does not provide task-level results, detailed evaluation procedures, or further evidence for interpreting the reported scores. The reported results concern the benchmark's 36 production MCP servers; the abstract does not establish whether they generalize to other environments.","speaker":{"name":"Jiang, Yizhen, Guo, Haowei, Bai, Tianyi, Hu, Zengjie, Ma, Lu, Zhao, Zhengyang, Sun, Haoze, Pei, Peng, Zhang, Wentao","roleAtSource":"Article author"},"source":{"title":"MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.07689","publishedAt":"2026-11-05","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce MCPHallu as a benchmark that builds reasoning, execution, and memory failure risks into tasks, rather than diagnosing failures only after execution.","excerpt":"MCPHallu, a benchmark for diagnosing hallucination-prone agent failures in MCP environments along three axes: reasoning, execution, and memory.","url":"https://arxiv.org/abs/2611.07689","relationship":"Source-stated"},{"text":"MCPHallu includes 358 tasks across five capability domains, run on 36 production MCP servers in isolated containers, with four targeted failure subtypes.","excerpt":"MCPHallu contains 358 tasks across five capability domains and runs on 36 production MCP servers in isolated containers.","url":"https://arxiv.org/abs/2611.07689","relationship":"Source-stated"},{"text":"In the authors' evaluation of fourteen agents from eight vendors across 5,012 trajectories, the average score was 0.642 on a scale where higher is better.","excerpt":"Current MCP agents remain vulnerable to hallucination-inducing tool environments, achieving an average score of 0.642 on a $[0,1]$ scale (higher is better).","url":"https://arxiv.org/abs/2611.07689","relationship":"Source-stated"},{"text":"The authors report lower scores on Unreachable Goal tasks and say agents often continue making tool calls or claim completion instead of recognizing infeasible goals.","excerpt":"the average score drops to 0.458 and 23.2\\% of runs pass, indicating that agents often continue issuing tool calls or claim successful completion rather than recognizing infeasible goals.","url":"https://arxiv.org/abs/2611.07689","relationship":"Source-stated"},{"text":"The authors report that model rankings differ across failure subtypes, arguing that aggregate task success alone may miss distinct weaknesses.","excerpt":"a model's rank shifts by 6.9 of 13 positions on average across subtypes","url":"https://arxiv.org/abs/2611.07689","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07689","checkedAt":"2026-11-08T03:18:06.988Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Would the subtype rankings and Unreachable Goal results hold in MCP environments beyond the 36 production servers tested?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents","summary":"The MCPHallu authors introduce a benchmark for testing how agents using the Model Context Protocol (MCP)—a system for discovering and invoking tools across servers—fail through reasoning, execution, and memory errors. They argue that existing benchmarks primarily measure task completion and, where they diagnose failures, do so after execution, while MCPHallu builds specific failure risks into tasks. Its 358 tasks run on 36 production MCP servers in isolated containers and target four subtypes: Branch Collapse, Unreachable Goal, Tool Misuse, and Context Forgetting.\n\nAcross 5,012 trajectories from fourteen agents by eight vendors, the authors report an average score of 0.642 on a scale where higher is better. Scores fall to 0.458 on Unreachable Goal tasks, and 23.2% of runs pass; the authors say agents often keep calling tools or claim success instead of recognizing infeasible goals. Model rankings also vary across subtypes, so the authors argue that aggregate task success can hide distinct weaknesses. The results describe performance on this benchmark; the abstract does not establish how well they predict behavior in other settings.","question":"Would the subtype rankings and Unreachable Goal results hold in MCP environments beyond the 36 production servers tested?","claims":[{"text":"The authors introduce MCPHallu as a benchmark that builds reasoning, execution, and memory failure risks into tasks, rather than diagnosing failures only after execution.","excerpt":"MCPHallu, a benchmark for diagnosing hallucination-prone agent failures in MCP environments along three axes: reasoning, execution, and memory.","support":"source-stated"},{"text":"MCPHallu includes 358 tasks across five capability domains, run on 36 production MCP servers in isolated containers, with four targeted failure subtypes.","excerpt":"MCPHallu contains 358 tasks across five capability domains and runs on 36 production MCP servers in isolated containers.","support":"source-stated"},{"text":"In the authors' evaluation of fourteen agents from eight vendors across 5,012 trajectories, the average score was 0.642 on a scale where higher is better.","excerpt":"Current MCP agents remain vulnerable to hallucination-inducing tool environments, achieving an average score of 0.642 on a $[0,1]$ scale (higher is better).","support":"source-stated"},{"text":"The authors report lower scores on Unreachable Goal tasks and say agents often continue making tool calls or claim completion instead of recognizing infeasible goals.","excerpt":"the average score drops to 0.458 and 23.2\\% of runs pass, indicating that agents often continue issuing tool calls or claim successful completion rather than recognizing infeasible goals.","support":"source-stated"},{"text":"The authors report that model rankings differ across failure subtypes, arguing that aggregate task success alone may miss distinct weaknesses.","excerpt":"a model's rank shifts by 6.9 of 13 positions on average across subtypes","support":"source-stated"}],"limitations":["The available evidence is the paper's abstract; it does not provide task-level results, detailed evaluation procedures, or further evidence for interpreting the reported scores.","The reported results concern the benchmark's 36 production MCP servers; the abstract does not establish whether they generalize to other environments."],"mediaId":"6da74d982fe142890f7a0039d75a1c18a012fd8ff5877b3401c90173ba248949","correctionReason":null,"id":"aaa37d04ead994ddf52270426680b3a707742939647a34207ccfd66d8c320412","publicationId":"f5ec3464-d9e1-4ba2-8761-56e27407f01e","version":1,"sourceUrl":"https://arxiv.org/abs/2611.07689","sourceTitle":"MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-05","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:07:17.485Z","createdAt":"2026-11-08T03:02:40.987Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"6da74d982fe142890f7a0039d75a1c18a012fd8ff5877b3401c90173ba248949","kind":"article","url":"https://arxiv.org/abs/2611.07689","sourceUrl":"https://arxiv.org/abs/2611.07689","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-08T02:33:05.492Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07689","checkedAt":"2026-11-08T03:18:06.988Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"6da74d982fe142890f7a0039d75a1c18a012fd8ff5877b3401c90173ba248949","confidence":0.9,"provider":"typesafe","model":"jev-1.13.0","inputHash":"fcd05c6af5a989ca8e4111360ebfc2c5f61c97945c9560ba9b3d8bd1e69dce78","decidedAt":"2026-11-08T02:33:06.261Z"},"provenance":{"textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observationId":"75c62cc85e746ba780b5cd7490a6f1dd9a98c5f4440eb71d196c17d212607d4f","retrievedAt":"2026-11-08T02:33:04.991Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"353e314f-ec31-4d7a-8bab-c4bae55fd75b"},"reviewer":{"model":"gpt-6-luna","runId":"be949a6c-0cdf-4234-8ce4-2b743440b40f"}},"audit":{"status":"passed","reviewedAt":"2026-11-08T03:02:40.987Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Jiang, Yizhen","url":null,"kind":"unknown"},{"name":"Guo, Haowei","url":null,"kind":"unknown"},{"name":"Bai, Tianyi","url":null,"kind":"unknown"},{"name":"Hu, Zengjie","url":null,"kind":"unknown"},{"name":"Ma, Lu","url":null,"kind":"unknown"},{"name":"Zhao, Zhengyang","url":null,"kind":"unknown"},{"name":"Sun, Haoze","url":null,"kind":"unknown"},{"name":"Pei, Peng","url":null,"kind":"unknown"},{"name":"Zhang, Wentao","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Jiang, Yizhen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Jiang, Yizhen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07689","locator":"head/meta[name=author|citation_author]"},"assertionId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.988Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","sourceUrl":"https://arxiv.org/abs/2611.07689","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z"}},{"actor":{"name":"Guo, Haowei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Guo, Haowei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07689","locator":"head/meta[name=author|citation_author]"},"assertionId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.988Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","sourceUrl":"https://arxiv.org/abs/2611.07689","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z"}},{"actor":{"name":"Bai, Tianyi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Bai, Tianyi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07689","locator":"head/meta[name=author|citation_author]"},"assertionId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.988Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","sourceUrl":"https://arxiv.org/abs/2611.07689","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z"}},{"actor":{"name":"Hu, Zengjie","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Hu, Zengjie","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07689","locator":"head/meta[name=author|citation_author]"},"assertionId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.988Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","sourceUrl":"https://arxiv.org/abs/2611.07689","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z"}},{"actor":{"name":"Ma, Lu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ma, Lu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07689","locator":"head/meta[name=author|citation_author]"},"assertionId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.988Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","sourceUrl":"https://arxiv.org/abs/2611.07689","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z"}},{"actor":{"name":"Zhao, Zhengyang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhao, Zhengyang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07689","locator":"head/meta[name=author|citation_author]"},"assertionId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.988Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","sourceUrl":"https://arxiv.org/abs/2611.07689","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z"}},{"actor":{"name":"Sun, Haoze","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sun, Haoze","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07689","locator":"head/meta[name=author|citation_author]"},"assertionId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.988Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","sourceUrl":"https://arxiv.org/abs/2611.07689","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z"}},{"actor":{"name":"Pei, Peng","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Pei, Peng","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07689","locator":"head/meta[name=author|citation_author]"},"assertionId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.988Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","sourceUrl":"https://arxiv.org/abs/2611.07689","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z"}},{"actor":{"name":"Zhang, Wentao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Wentao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07689","locator":"head/meta[name=author|citation_author]"},"assertionId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e:participation:8","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.988Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","sourceUrl":"https://arxiv.org/abs/2611.07689","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z"}}],"participationCoverage":{"total":9,"returned":9,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.07689","snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z","basis":"publisher-page-metadata","sourceLinks":48,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/aaa37d04ead994ddf52270426680b3a707742939647a34207ccfd66d8c320412/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-mcphallu-benchmarking-reasoning-execution-and-aaa37d04","formats":{"html":"/readings/arxiv-cs-ai-mcphallu-benchmarking-reasoning-execution-and-aaa37d04","markdown":"/readings/arxiv-cs-ai-mcphallu-benchmarking-reasoning-execution-and-aaa37d04.md","json":"/readings/arxiv-cs-ai-mcphallu-benchmarking-reasoning-execution-and-aaa37d04.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-hallucination","name":"Hallucination","explanation":"When an AI system states something false or made up as if it were true.","referenceUrl":"https://developers.google.com/machine-learning/glossary#hallucination"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"aaa37d04ead994ddf52270426680b3a707742939647a34207ccfd66d8c320412","revision":"f5ec3464-d9e1-4ba2-8761-56e27407f01e","revisionKind":"publication-id","title":"arXiv cs.AI: MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents","url":"https://escapemeridian.com/readings/arxiv-cs-ai-mcphallu-benchmarking-reasoning-execution-and-aaa37d04"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.07689","revision":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","revisionKind":"sha256-source-text","title":"MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents","url":"https://arxiv.org/abs/2611.07689"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"aaa37d04ead994ddf52270426680b3a707742939647a34207ccfd66d8c320412","title":"arXiv cs.AI: MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents","argument":"The MCPHallu authors introduce a benchmark for testing how agents using the Model Context Protocol (MCP)—a system for discovering and invoking tools across servers—fail through reasoning, execution, and memory errors. They argue that existing benchmarks primarily measure task completion and, where they diagnose failures, do so after execution, while MCPHallu builds specific failure risks into tasks. Its 358 tasks run on 36 production MCP servers in isolated containers and target four subtypes: Branch Collapse, Unreachable Goal, Tool Misuse, and Context Forgetting.\n\nAcross 5,012 trajectories from fourteen agents by eight vendors, the authors report an average score of 0.642 on a scale where higher is better. Scores fall to 0.458 on Unreachable Goal tasks, and 23.2% of runs pass; the authors say agents often keep calling tools or claim success instead of recognizing infeasible goals. Model rankings also vary across subtypes, so the authors argue that aggregate task success can hide distinct weaknesses. The results describe performance on this benchmark; the abstract does not establish how well they predict behavior in other settings.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The available evidence is the paper's abstract; it does not provide task-level results, detailed evaluation procedures, or further evidence for interpreting the reported scores. The reported results concern the benchmark's 36 production MCP servers; the abstract does not establish whether they generalize to other environments.","speaker":{"name":"Jiang, Yizhen, Guo, Haowei, Bai, Tianyi, Hu, Zengjie, Ma, Lu, Zhao, Zhengyang, Sun, Haoze, Pei, Peng, Zhang, Wentao","roleAtSource":"Article author"},"source":{"title":"MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.07689","publishedAt":"2026-11-05","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce MCPHallu as a benchmark that builds reasoning, execution, and memory failure risks into tasks, rather than diagnosing failures only after execution.","excerpt":"MCPHallu, a benchmark for diagnosing hallucination-prone agent failures in MCP environments along three axes: reasoning, execution, and memory.","url":"https://arxiv.org/abs/2611.07689","relationship":"Source-stated"},{"text":"MCPHallu includes 358 tasks across five capability domains, run on 36 production MCP servers in isolated containers, with four targeted failure subtypes.","excerpt":"MCPHallu contains 358 tasks across five capability domains and runs on 36 production MCP servers in isolated containers.","url":"https://arxiv.org/abs/2611.07689","relationship":"Source-stated"},{"text":"In the authors' evaluation of fourteen agents from eight vendors across 5,012 trajectories, the average score was 0.642 on a scale where higher is better.","excerpt":"Current MCP agents remain vulnerable to hallucination-inducing tool environments, achieving an average score of 0.642 on a $[0,1]$ scale (higher is better).","url":"https://arxiv.org/abs/2611.07689","relationship":"Source-stated"},{"text":"The authors report lower scores on Unreachable Goal tasks and say agents often continue making tool calls or claim completion instead of recognizing infeasible goals.","excerpt":"the average score drops to 0.458 and 23.2\\% of runs pass, indicating that agents often continue issuing tool calls or claim successful completion rather than recognizing infeasible goals.","url":"https://arxiv.org/abs/2611.07689","relationship":"Source-stated"},{"text":"The authors report that model rankings differ across failure subtypes, arguing that aggregate task success alone may miss distinct weaknesses.","excerpt":"a model's rank shifts by 6.9 of 13 positions on average across subtypes","url":"https://arxiv.org/abs/2611.07689","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07689","checkedAt":"2026-11-08T03:18:06.988Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Would the subtype rankings and Unreachable Goal results hold in MCP environments beyond the 36 production servers tested?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents","summary":"The MCPHallu authors introduce a benchmark for testing how agents using the Model Context Protocol (MCP)—a system for discovering and invoking tools across servers—fail through reasoning, execution, and memory errors. They argue that existing benchmarks primarily measure task completion and, where they diagnose failures, do so after execution, while MCPHallu builds specific failure risks into tasks. Its 358 tasks run on 36 production MCP servers in isolated containers and target four subtypes: Branch Collapse, Unreachable Goal, Tool Misuse, and Context Forgetting.\n\nAcross 5,012 trajectories from fourteen agents by eight vendors, the authors report an average score of 0.642 on a scale where higher is better. Scores fall to 0.458 on Unreachable Goal tasks, and 23.2% of runs pass; the authors say agents often keep calling tools or claim success instead of recognizing infeasible goals. Model rankings also vary across subtypes, so the authors argue that aggregate task success can hide distinct weaknesses. The results describe performance on this benchmark; the abstract does not establish how well they predict behavior in other settings.","question":"Would the subtype rankings and Unreachable Goal results hold in MCP environments beyond the 36 production servers tested?","claims":[{"text":"The authors introduce MCPHallu as a benchmark that builds reasoning, execution, and memory failure risks into tasks, rather than diagnosing failures only after execution.","excerpt":"MCPHallu, a benchmark for diagnosing hallucination-prone agent failures in MCP environments along three axes: reasoning, execution, and memory.","support":"source-stated"},{"text":"MCPHallu includes 358 tasks across five capability domains, run on 36 production MCP servers in isolated containers, with four targeted failure subtypes.","excerpt":"MCPHallu contains 358 tasks across five capability domains and runs on 36 production MCP servers in isolated containers.","support":"source-stated"},{"text":"In the authors' evaluation of fourteen agents from eight vendors across 5,012 trajectories, the average score was 0.642 on a scale where higher is better.","excerpt":"Current MCP agents remain vulnerable to hallucination-inducing tool environments, achieving an average score of 0.642 on a $[0,1]$ scale (higher is better).","support":"source-stated"},{"text":"The authors report lower scores on Unreachable Goal tasks and say agents often continue making tool calls or claim completion instead of recognizing infeasible goals.","excerpt":"the average score drops to 0.458 and 23.2\\% of runs pass, indicating that agents often continue issuing tool calls or claim successful completion rather than recognizing infeasible goals.","support":"source-stated"},{"text":"The authors report that model rankings differ across failure subtypes, arguing that aggregate task success alone may miss distinct weaknesses.","excerpt":"a model's rank shifts by 6.9 of 13 positions on average across subtypes","support":"source-stated"}],"limitations":["The available evidence is the paper's abstract; it does not provide task-level results, detailed evaluation procedures, or further evidence for interpreting the reported scores.","The reported results concern the benchmark's 36 production MCP servers; the abstract does not establish whether they generalize to other environments."],"mediaId":"6da74d982fe142890f7a0039d75a1c18a012fd8ff5877b3401c90173ba248949","correctionReason":null,"id":"aaa37d04ead994ddf52270426680b3a707742939647a34207ccfd66d8c320412","publicationId":"f5ec3464-d9e1-4ba2-8761-56e27407f01e","version":1,"sourceUrl":"https://arxiv.org/abs/2611.07689","sourceTitle":"MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-05","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:07:17.485Z","createdAt":"2026-11-08T03:02:40.987Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"6da74d982fe142890f7a0039d75a1c18a012fd8ff5877b3401c90173ba248949","kind":"article","url":"https://arxiv.org/abs/2611.07689","sourceUrl":"https://arxiv.org/abs/2611.07689","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-08T02:33:05.492Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07689","checkedAt":"2026-11-08T03:18:06.988Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"6da74d982fe142890f7a0039d75a1c18a012fd8ff5877b3401c90173ba248949","confidence":0.9,"provider":"typesafe","model":"jev-1.13.0","inputHash":"fcd05c6af5a989ca8e4111360ebfc2c5f61c97945c9560ba9b3d8bd1e69dce78","decidedAt":"2026-11-08T02:33:06.261Z"},"provenance":{"textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observationId":"75c62cc85e746ba780b5cd7490a6f1dd9a98c5f4440eb71d196c17d212607d4f","retrievedAt":"2026-11-08T02:33:04.991Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"353e314f-ec31-4d7a-8bab-c4bae55fd75b"},"reviewer":{"model":"gpt-6-luna","runId":"be949a6c-0cdf-4234-8ce4-2b743440b40f"}},"audit":{"status":"passed","reviewedAt":"2026-11-08T03:02:40.987Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Jiang, Yizhen","url":null,"kind":"unknown"},{"name":"Guo, Haowei","url":null,"kind":"unknown"},{"name":"Bai, Tianyi","url":null,"kind":"unknown"},{"name":"Hu, Zengjie","url":null,"kind":"unknown"},{"name":"Ma, Lu","url":null,"kind":"unknown"},{"name":"Zhao, Zhengyang","url":null,"kind":"unknown"},{"name":"Sun, Haoze","url":null,"kind":"unknown"},{"name":"Pei, Peng","url":null,"kind":"unknown"},{"name":"Zhang, Wentao","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Jiang, Yizhen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Jiang, Yizhen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07689","locator":"head/meta[name=author|citation_author]"},"assertionId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.988Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","sourceUrl":"https://arxiv.org/abs/2611.07689","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z"}},{"actor":{"name":"Guo, Haowei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Guo, Haowei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07689","locator":"head/meta[name=author|citation_author]"},"assertionId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.988Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","sourceUrl":"https://arxiv.org/abs/2611.07689","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z"}},{"actor":{"name":"Bai, Tianyi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Bai, Tianyi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07689","locator":"head/meta[name=author|citation_author]"},"assertionId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.988Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","sourceUrl":"https://arxiv.org/abs/2611.07689","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z"}},{"actor":{"name":"Hu, Zengjie","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Hu, Zengjie","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07689","locator":"head/meta[name=author|citation_author]"},"assertionId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.988Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","sourceUrl":"https://arxiv.org/abs/2611.07689","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z"}},{"actor":{"name":"Ma, Lu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ma, Lu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07689","locator":"head/meta[name=author|citation_author]"},"assertionId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.988Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","sourceUrl":"https://arxiv.org/abs/2611.07689","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z"}},{"actor":{"name":"Zhao, Zhengyang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhao, Zhengyang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07689","locator":"head/meta[name=author|citation_author]"},"assertionId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.988Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","sourceUrl":"https://arxiv.org/abs/2611.07689","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z"}},{"actor":{"name":"Sun, Haoze","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sun, Haoze","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07689","locator":"head/meta[name=author|citation_author]"},"assertionId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.988Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","sourceUrl":"https://arxiv.org/abs/2611.07689","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z"}},{"actor":{"name":"Pei, Peng","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Pei, Peng","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07689","locator":"head/meta[name=author|citation_author]"},"assertionId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.988Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","sourceUrl":"https://arxiv.org/abs/2611.07689","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z"}},{"actor":{"name":"Zhang, Wentao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Wentao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07689","locator":"head/meta[name=author|citation_author]"},"assertionId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e:participation:8","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T03:18:06.988Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","sourceUrl":"https://arxiv.org/abs/2611.07689","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z"}}],"participationCoverage":{"total":9,"returned":9,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.07689","snapshotId":"c41541c849bc65cd57df523a13e0c5c1961ac4521adc5b9059026d46a0fbb71e","contentSha256":"33c36e8dfd8a719013f3e5f20b1ba67118149ca587c9a8e1257e5da7ba41377c","textHash":"4970e3870a56a91cb19aa642bab73ce8f16b2f7d5d3c47898523795a0f9bd75b","observedAt":"2026-11-08T03:18:06.988Z","basis":"publisher-page-metadata","sourceLinks":48,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/aaa37d04ead994ddf52270426680b3a707742939647a34207ccfd66d8c320412/source"}}}],"audits":[{"publicationId":"f5ec3464-d9e1-4ba2-8761-56e27407f01e","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the benchmark, its design rationale and the results to the MCPHallu authors, with arXiv cs.AI as the publisher. Findings stay framed as the authors' reports and arguments rather than established fact.","excerpt":"We introduce MCPHallu, a benchmark for diagnosing hallucination-prone agent failures in MCP environments along three axes: reasoning, execution, and memory."},{"dimension":"summary","status":"pass","explanation":"The summary leads with the actor and the development, and explains MCP in plain terms. It conveys the core finding: agents keep calling tools or claim success on infeasible goals, and rankings shift by subtype. The abstract does not define Branch Collapse, Tool Misuse, Context Forgetting or 'trajectories', so leaving them unexplained is acceptable; tooltips could help. Calling the rankings 'vary' slightly softens the source's 'diverge sharply' but does not change its meaning. The question follows from the stated generalization gap.","excerpt":"Subtype rankings diverge sharply: a model's rank shifts by 6.9 of 13 positions on average across subtypes"},{"dimension":"date","status":"pass","explanation":"The originalPublished date of 2026-11-05 matches the arXiv v1 submission date. The feed date of 2026-11-06 is later and is correctly not used.","excerpt":"[Submitted on 5 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"The counts (358 tasks, 36 servers, fourteen agents, eight vendors, 5,012 trajectories) and the scores 0.642, 0.458 and 23.2% match the abstract. The point that aggregate success can hide weaknesses is a reasonable restatement of the subtype-level design rationale and the reported rank divergence. The limitations about generalization are appropriately conditional and invent no benefits or costs.","excerpt":"This design enables subtype-level analysis rather than aggregate task success alone."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct arXiv abstract page for this paper. It is the most direct way to reach the source.","excerpt":null}],"createdAt":"2026-11-08T03:02:40.987Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}