Escape Meridian

Gupta, Sachin · Nov 16, 2026

AgentFaultBench: State-Verified Evaluation of Tool-Fault Recovery

Markdown · JSON · Original source

Sachin Gupta’s AgentFaultBench tests whether recovery advice helps LangGraph and CrewAI agents finish specified tasks after tool failures. In an isolated GitLab installation, it checks resulting files and issues, required reads and verification, and the final response. Prompts provide expected issue and report text, so Gupta says the tests execution rather than content generation. In the primary comparison, advice did not consistently improve success over a neutral reminder: changes were -8.33, 0.00 and +1.06 percentage points, with missing outcomes making two comparisons inconclusive. Later experiments with a revised policy found improvement for GPT-4o mini, while Sol and already completed the recoverable cases. On six related tasks not previously used by that policy, guided runs had more correct final states than neutral-feedback runs. The reported gain was +9.90 to +10.42 percentage points when missing outcomes were included. Removing issue IDs resulted in 47/48 successes versus 46/48 with IDs; other execution information remained. A second checker agreed with observed primary outcomes and constructed test cases, but no human validation was collected. Gupta’s conclusion is methodological: test recovery advice against explicit recovery controls rather than assume it helps.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

arXiv cs.AIArticle · arxiv.orgOpen original source

Original source

cs.AI · Article

Source details

Original source date: Nov 16, 2026 (2026-11-16).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-17T09:17:45.965Z. First observed: 2026-11-17T08:31:15.381Z. Retrieved: 2026-11-17T09:16:42.973Z.

Referenced by

Recorded citations in public readings. A reference alone does not establish agreement or influence.

No public references recorded here yet.

Reference data (JSON)

Limits

The tasks provide expected issue and report text, so the paper describes them as tests of execution rather than content generation. The abstract reports no human validation. The primary comparison has missing outcomes, leaving two comparisons inconclusive. The evidence summarized here describes specified tasks and agent setups; it does not establish how results transfer beyond them.

Open question

How would the revised recovery policy perform on tasks beyond the six related tasks tested, and with human validation?

Source-stated

AgentFaultBench evaluates LangGraph and CrewAI agents on specified tasks in an isolated GitLab installation, checking task state and required actions as well as the final response.

AgentFaultBench evaluates LangGraph and CrewAI agents performing specified tasks in an isolated GitLab installation. We inject tool errors and check both the resulting files and issues and the required reads, verification and final response.

Source-stated

The primary comparison found mixed changes relative to a neutral reminder; missing outcomes make two comparisons inconclusive.

In the primary comparison of 1,944 planned runs, advice changes success by -8.33, 0.00 and +1.06 percentage points relative to a neutral reminder; missing outcomes leave the latter two comparisons inconclusive.

Source-stated

In a separate study on six related tasks, the guided condition had more correct final states than neutral feedback, with the stated gain accounting for missing outcomes.

In a separate 624-run study on six related tasks not previously used by that policy, the guided condition records 161 correct final states and one unknown outcome in 192 planned runs, versus 142/192 with neutral feedback. The gain lies between +9.90 and +10.42 pp when missing outcomes are included.

Source-stated

The abstract reports no human validation and argues for testing recovery advice against explicit controls rather than assuming it helps.

No human validation was collected. The findings support testing advice against explicit recovery controls rather than assuming it helps.

Source and contributor context

Gupta, Sachin · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.

Source media

No suitable media was found by the supported page extraction.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

Claude

Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.

Editorial provenance

Version 1. Publication ID: 92605c70-1321-4189-8fbd-969c1902f39b.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-17T09:17:45.965Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material_development.

Observation: b64723e619f3de32972cfd767c251e03e8742167d32ce1cfd697fae5b40dc680. Source text hash: 877d5e83e0040a0ba44db3bf4542ecf87c37d68d6e0835cd0471f7d63c35197f.

Revision and source-check history

Version 1 · 2026-11-17T09:17:45.965Z

AgentFaultBench: State-Verified Evaluation of Tool-Fault Recovery

Sachin Gupta’s AgentFaultBench tests whether recovery advice helps LangGraph and CrewAI agents finish specified tasks after tool failures. In an isolated GitLab installation, it checks resulting files and issues, required reads and verification, and the final response. Prompts provide expected issue and report text, so Gupta says the tests execution rather than content generation. In the primary comparison, advice did not consistently improve success over a neutral reminder: changes were -8.33, 0.00 and +1.06 percentage points, with missing outcomes making two comparisons inconclusive. Later experiments with a revised policy found improvement for GPT-4o mini, while Sol and already completed the recoverable cases. On six related tasks not previously used by that policy, guided runs had more correct final states than neutral-feedback runs. The reported gain was +9.90 to +10.42 percentage points when missing outcomes were included. Removing issue IDs resulted in 47/48 successes versus 46/48 with IDs; other execution information remained. A second checker agreed with observed primary outcomes and constructed test cases, but no human validation was collected. Gupta’s conclusion is methodological: test recovery advice against explicit recovery controls rather than assume it helps.

Original source

cs.AI · Article

Source details

Original source date: Nov 16, 2026 (2026-11-16).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-17T09:17:45.965Z. First observed: 2026-11-17T08:31:15.381Z. Retrieved: 2026-11-17T09:16:42.973Z.

Limits

The tasks provide expected issue and report text, so the paper describes them as tests of execution rather than content generation. The abstract reports no human validation. The primary comparison has missing outcomes, leaving two comparisons inconclusive. The evidence summarized here describes specified tasks and agent setups; it does not establish how results transfer beyond them.

Open question

How would the revised recovery policy perform on tasks beyond the six related tasks tested, and with human validation?

Source-stated

AgentFaultBench evaluates LangGraph and CrewAI agents on specified tasks in an isolated GitLab installation, checking task state and required actions as well as the final response.

AgentFaultBench evaluates LangGraph and CrewAI agents performing specified tasks in an isolated GitLab installation. We inject tool errors and check both the resulting files and issues and the required reads, verification and final response.

Source-stated

The primary comparison found mixed changes relative to a neutral reminder; missing outcomes make two comparisons inconclusive.

In the primary comparison of 1,944 planned runs, advice changes success by -8.33, 0.00 and +1.06 percentage points relative to a neutral reminder; missing outcomes leave the latter two comparisons inconclusive.

Source-stated

In a separate study on six related tasks, the guided condition had more correct final states than neutral feedback, with the stated gain accounting for missing outcomes.

In a separate 624-run study on six related tasks not previously used by that policy, the guided condition records 161 correct final states and one unknown outcome in 192 planned runs, versus 142/192 with neutral feedback. The gain lies between +9.90 and +10.42 pp when missing outcomes are included.

Source-stated

The abstract reports no human validation and argues for testing recovery advice against explicit controls rather than assuming it helps.

No human validation was collected. The findings support testing advice against explicit recovery controls rather than assuming it helps.

Source and contributor context

Gupta, Sachin · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.

Source media

No suitable media was found by the supported page extraction.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

Claude

Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.

Editorial provenance

Version 1. Publication ID: 92605c70-1321-4189-8fbd-969c1902f39b.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-17T09:17:45.965Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material_development.

Observation: b64723e619f3de32972cfd767c251e03e8742167d32ce1cfd697fae5b40dc680. Source text hash: 877d5e83e0040a0ba44db3bf4542ecf87c37d68d6e0835cd0471f7d63c35197f.

passed · 2026-11-17T09:17:45.965Z · anthropic/claude-opus-5.5

Publication: 92605c70-1321-4189-8fbd-969c1902f39b

attribution — pass: The reading correctly attributes AgentFaultBench and its conclusions to author Sachin Gupta, the sole listed author of the arXiv paper. Findings are framed as the paper's reported results rather than established facts.

Authors:Sachin Gupta

summary — pass: The summary leads with the actor, the question, and the setup. It preserves the mixed primary result, the later gain, the missing-outcome caveats, the lack of human validation, and the methodological conclusion. The question is a substantive open issue. Minor tooltip candidates include LangGraph, CrewAI, and Sol. The issue-ID ablation sentence is terse, but neither problem blocks a general reader's understanding of the central development.

Does recovery advice help an agent finish a task after a tool fails?

date — pass: originalPublished is 2026-11-16, which matches the v1 submission date on arXiv. The 2026-11-17 date is only the later feed entry.

[Submitted on 16 Nov 2026]

support — pass: All figures match the abstract: -8.33, 0.00 and +1.06 pp; the +9.90 to +10.42 pp gain; 47/48 vs 46/48; and the agreement of a second checker. The reading also states correctly that two comparisons are inconclusive, that no human validation was collected, and that the conclusion is methodological. The characterization 'did not consistently improve' is a fair reading of the mixed results. No unsupported benefits are implied.

advice changes success by -8.33, 0.00 and +1.06 percentage points relative to a neutral reminder; missing outcomes leave the latter two comparisons inconclusive.

media — pass: The selected media is the verified direct link to the arXiv abstract page. It is the original source and the most relevant access point.

Challenge or correct this reading

Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.