# BABA-Bench’s authors argue patient-advocacy agents must follow rules, not just reach outcomes [Canonical reading](https://escapemeridian.com/readings/baba-bench-s-authors-argue-patient-advocacy-agents-must-follow-45d1985c) The authors of BABA-Bench argue that an AI agent helping a patient must achieve the right outcome and follow the required procedure—for example, get permission before sharing records. Their benchmark contains 47 interactive worlds based on de-identified patient-advocacy cases reviewed by expert advocates. Agents must uncover information and coordinate with insurers, clinics, suppliers, pharmacies, and community organizations; the benchmark checks each action for permission, supporting evidence, and prohibited conduct. Across 1,974 runs, the authors report that agents sometimes reached all required outcomes by an invalid route: 254 of the 921 successful runs failed procedural checks. Claude Opus 5 reached all required outcomes in 81.6% of its runs, but passed all checks in 52.5%. When two models received a verified route, their pass rates rose substantially. The authors conclude that, for those models, finding a route—not executing one—is the main problem. The abstract says they evaluated 14 frontier models. Escape Meridian interpretation: in patient advocacy, a favorable result alone may not show that an agent respected permission and evidence requirements. The benchmark tests those requirements in simulated worlds; the abstract does not establish how its results translate to real patient care. How well do the benchmark’s interactive worlds capture the constraints agents face Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Oct 29, 2026 (2026-10-29). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-07T07:46:11.295Z. First observed: 2026-11-02T05:06:22.143Z. Retrieved: 2026-11-02T11:47:35.456Z. - [BABA-Bench: Evaluating Long-Horizon Patient-Advocacy Agents](https://arxiv.org/abs/2610.41076) ## Limits The evidence is an arXiv abstract; it does not provide detailed benchmark methods, case composition, or results beyond those summarized there. The abstract reports results in benchmark worlds and does not establish how they translate to real patient care. The evidence does not specify which two models received verified routes. ## Open question How well do BABA-Bench’s interactive worlds capture the constraints agents would face in actual patient-advocacy cases? ## Source-stated The authors say patient-advocacy agents need both to reach the correct outcome and to follow the necessary procedure, including obtaining permission before sharing records. > It must reach the correct outcome, and it must follow the necessary procedure. For example, the agent must get the patient's permission before it shares the patient's records. - [Supporting source](https://arxiv.org/abs/2610.41076) ## Source-stated BABA-Bench comprises 47 interactive worlds built from de-identified patient-advocacy cases reviewed by expert advocates; agents must find hidden information and coordinate across organizations. > We introduce BABA-Bench, a benchmark of 47 interactive worlds. We built each world from a de-identified, real patient advocacy case that expert advocates reviewed. - [Supporting source](https://arxiv.org/abs/2610.41076) ## Source-stated The benchmark checks each action for permission, evidence, and whether the action is prohibited. > The engine checks each action at the time the agent does it. It checks if the agent has permission, if the agent has evidence for the action, and if the action is prohibited. - [Supporting source](https://arxiv.org/abs/2610.41076) ## Source-stated Across 1,974 runs, 254 of the 921 runs reaching all required outcomes did not follow the necessary procedure. > Of the 921 runs that reached all required outcomes, 254 (28%) did not follow the necessary procedure. - [Supporting source](https://arxiv.org/abs/2610.41076) ## Source-stated Claude Opus 5 reached all required outcomes in 81.6% of runs but passed all checks in 52.5%. > Claude Opus 5 reached all required outcomes in 81.6% of its runs, but only 52.5% of its runs passed all checks. - [Supporting source](https://arxiv.org/abs/2610.41076) ## Source-stated With a verified route, two models' pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%; the authors say route-finding is the main problem for these models. > When we gave two models a verified route, their pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%. For these models, the main problem is to find a route, not to execute it. - [Supporting source](https://arxiv.org/abs/2610.41076) ## Escape Meridian interpretation A favorable outcome alone may not establish that a patient-advocacy agent respected required permissions and evidence checks; this is an interpretation of the benchmark's distinction between outcome and procedure. > It must reach the correct outcome, and it must follow the necessary procedure. - [Supporting source](https://arxiv.org/abs/2610.41076) ## Source and contributor context Vasudevan, Shriram, Mehta, Videet, Sweeney, Connor · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## AI agent An AI system that chooses and carries out actions toward a goal, often using software tools. - [Definition reference](https://www.ibm.com/think/topics/ai-agents) ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Frontier model One of the most capable AI models available at a given time; the set shifts as models improve. - [Definition reference](https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Claude Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding. - [Explore this context](/entities/claude) - [Reference](https://claude.com/product/overview) - [Reference](https://www.anthropic.com/news) - [Reference](https://www.anthropic.com/research/team/economics) ## Editorial provenance Version 1. Publication ID: 6feb3bc9-51e0-4a18-8f96-877025eff483. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-07T07:46:11.295Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: fbb2be26796d960eb030afaa56de5391d83d8d147287059c65892a82de3b5a53. Source text hash: 3e74d06c533e049e6d0af5a58a69a0f103a85d428c598307049d0b957f57a2fc. ## Revision and source-check history ### Version 1 — 2026-11-07T07:46:11.295Z The authors of BABA-Bench argue that an AI agent helping a patient must achieve the right outcome and follow the required procedure—for example, get permission before sharing records. Their benchmark contains 47 interactive worlds based on de-identified patient-advocacy cases reviewed by expert advocates. Agents must uncover information and coordinate with insurers, clinics, suppliers, pharmacies, and community organizations; the benchmark checks each action for permission, supporting evidence, and prohibited conduct. Across 1,974 runs, the authors report that agents sometimes reached all required outcomes by an invalid route: 254 of the 921 successful runs failed procedural checks. Claude Opus 5 reached all required outcomes in 81.6% of its runs, but passed all checks in 52.5%. When two models received a verified route, their pass rates rose substantially. The authors conclude that, for those models, finding a route—not executing one—is the main problem. The abstract says they evaluated 14 frontier models. Escape Meridian interpretation: in patient advocacy, a favorable result alone may not show that an agent respected permission and evidence requirements. The benchmark tests those requirements in simulated worlds; the abstract does not establish how its results translate to real patient care. How well do the benchmark’s interactive worlds capture the constraints agents face ## Original source arXiv cs.AI · Article Original source date: Oct 29, 2026 (2026-10-29). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-07T07:46:11.295Z. First observed: 2026-11-02T05:06:22.143Z. Retrieved: 2026-11-02T11:47:35.456Z. - [BABA-Bench: Evaluating Long-Horizon Patient-Advocacy Agents](https://arxiv.org/abs/2610.41076) ## Limits The evidence is an arXiv abstract; it does not provide detailed benchmark methods, case composition, or results beyond those summarized there. The abstract reports results in benchmark worlds and does not establish how they translate to real patient care. The evidence does not specify which two models received verified routes. ## Open question How well do BABA-Bench’s interactive worlds capture the constraints agents would face in actual patient-advocacy cases? ## Source-stated The authors say patient-advocacy agents need both to reach the correct outcome and to follow the necessary procedure, including obtaining permission before sharing records. > It must reach the correct outcome, and it must follow the necessary procedure. For example, the agent must get the patient's permission before it shares the patient's records. - [Supporting source](https://arxiv.org/abs/2610.41076) ## Source-stated BABA-Bench comprises 47 interactive worlds built from de-identified patient-advocacy cases reviewed by expert advocates; agents must find hidden information and coordinate across organizations. > We introduce BABA-Bench, a benchmark of 47 interactive worlds. We built each world from a de-identified, real patient advocacy case that expert advocates reviewed. - [Supporting source](https://arxiv.org/abs/2610.41076) ## Source-stated The benchmark checks each action for permission, evidence, and whether the action is prohibited. > The engine checks each action at the time the agent does it. It checks if the agent has permission, if the agent has evidence for the action, and if the action is prohibited. - [Supporting source](https://arxiv.org/abs/2610.41076) ## Source-stated Across 1,974 runs, 254 of the 921 runs reaching all required outcomes did not follow the necessary procedure. > Of the 921 runs that reached all required outcomes, 254 (28%) did not follow the necessary procedure. - [Supporting source](https://arxiv.org/abs/2610.41076) ## Source-stated Claude Opus 5 reached all required outcomes in 81.6% of runs but passed all checks in 52.5%. > Claude Opus 5 reached all required outcomes in 81.6% of its runs, but only 52.5% of its runs passed all checks. - [Supporting source](https://arxiv.org/abs/2610.41076) ## Source-stated With a verified route, two models' pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%; the authors say route-finding is the main problem for these models. > When we gave two models a verified route, their pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%. For these models, the main problem is to find a route, not to execute it. - [Supporting source](https://arxiv.org/abs/2610.41076) ## Escape Meridian interpretation A favorable outcome alone may not establish that a patient-advocacy agent respected required permissions and evidence checks; this is an interpretation of the benchmark's distinction between outcome and procedure. > It must reach the correct outcome, and it must follow the necessary procedure. - [Supporting source](https://arxiv.org/abs/2610.41076) ## Source and contributor context Vasudevan, Shriram, Mehta, Videet, Sweeney, Connor · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## AI agent An AI system that chooses and carries out actions toward a goal, often using software tools. - [Definition reference](https://www.ibm.com/think/topics/ai-agents) ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Frontier model One of the most capable AI models available at a given time; the set shifts as models improve. - [Definition reference](https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Claude Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding. - [Explore this context](/entities/claude) - [Reference](https://claude.com/product/overview) - [Reference](https://www.anthropic.com/news) - [Reference](https://www.anthropic.com/research/team/economics) ## Editorial provenance Version 1. Publication ID: 6feb3bc9-51e0-4a18-8f96-877025eff483. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-07T07:46:11.295Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: fbb2be26796d960eb030afaa56de5391d83d8d147287059c65892a82de3b5a53. Source text hash: 3e74d06c533e049e6d0af5a58a69a0f103a85d428c598307049d0b957f57a2fc. ### uncertain — 2026-11-07T07:46:11.295Z — anthropic/claude-opus-5.5 Publication: 6feb3bc9-51e0-4a18-8f96-877025eff483 attribution — pass: The reading consistently credits BABA-Bench's authors (Vasudevan, Mehta, Sweeney) and keeps the findings as their reported results. The Escape Meridian interpretation is clearly labeled. > Authors:Shriram Vasudevan, Videet Mehta, Connor Sweeney summary — uncertain: First, the summary's final paragraph ends with a truncated, unpunctuated fragment ('How well do the benchmark's interactive worlds capture the constraints agents face'). Smallest repair: delete it, since the question field already covers it. Second, the draft says agents 'sometimes' reached outcomes by an invalid route, but the source says 'often', which softens the authors' central finding. Restore 'often'. Third, the sentence about 14 frontier models sits awkwardly after the conclusion; move it next to the 1,974 runs. Otherwise the summary is understandable to a general reader, and the question follows from the evidence. > Agents often reached the correct outcome by an invalid route. date — pass: originalPublished 2026-10-29 matches the arXiv v1 submission date. The feed date of 2026-11-02 is later and correctly not used. > \[Submitted on 29 Oct 2026\] support — pass: All numeric claims match the abstract: 47 worlds, 14 models, 1,974 runs, 254 of 921, 81.6% versus 52.5%, and 29.8% to 91.5% and 44.0% to 97.2%. The interpretation is labeled and conditional. The limitations correctly state that real-world translation is not established and that the two models are not named. > When we gave two models a verified route, their pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%. media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/45d1985c0e00d4f2ba3e9616735c8eff18206dbd9179ae191b56094fb186cc75/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/baba-bench-s-authors-argue-patient-advocacy-agents-must-follow-45d1985c.json)