Vasudevan, Shriram, Mehta, Videet, Sweeney, Connor · Oct 29, 2026
BABA-Bench’s authors argue patient-advocacy agents must follow rules, not just reach outcomes
Markdown · JSON · Original source
The authors of BABA-Bench argue that an helping a patient must achieve the right outcome and follow the required procedure—for example, get permission before sharing records. Their contains 47 interactive worlds based on de-identified patient-advocacy cases reviewed by expert advocates. Agents must uncover information and coordinate with insurers, clinics, suppliers, pharmacies, and community organizations; the benchmark checks each action for permission, supporting evidence, and prohibited conduct. Across 1,974 runs, the authors report that agents sometimes reached all required outcomes by an invalid route: 254 of the 921 successful runs failed procedural checks. Opus 5 reached all required outcomes in 81.6% of its runs, but passed all checks in 52.5%. When two models received a verified route, their pass rates rose substantially. The authors conclude that, for those models, finding a route—not executing one—is the main problem. The abstract says they evaluated 14 . Escape Meridian interpretation: in patient advocacy, a favorable result alone may not show that an agent respected permission and evidence requirements. The benchmark tests those requirements in simulated worlds; the abstract does not establish how its results translate to real patient care. How well do the benchmark’s interactive worlds capture the constraints agents face
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 29, 2026 (2026-10-29).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-07T07:46:11.295Z. First observed: 2026-11-02T05:06:22.143Z. Retrieved: 2026-11-02T11:47:35.456Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The evidence is an abstract; it does not provide detailed methods, case composition, or results beyond those summarized there. The abstract reports results in benchmark worlds and does not establish how they translate to real patient care. The evidence does not specify which two models received verified routes.
Open question
How well do BABA-Bench’s interactive worlds capture the constraints agents would face in actual patient-advocacy cases?
Source-stated
The authors say patient-advocacy agents need both to reach the correct outcome and to follow the necessary procedure, including obtaining permission before sharing records.
It must reach the correct outcome, and it must follow the necessary procedure. For example, the agent must get the patient's permission before it shares the patient's records.
Source-stated
BABA-Bench comprises 47 interactive worlds built from de-identified patient-advocacy cases reviewed by expert advocates; agents must find hidden information and coordinate across organizations.
We introduce BABA-Bench, a benchmark of 47 interactive worlds. We built each world from a de-identified, real patient advocacy case that expert advocates reviewed.
Source-stated
The benchmark checks each action for permission, evidence, and whether the action is prohibited.
The engine checks each action at the time the agent does it. It checks if the agent has permission, if the agent has evidence for the action, and if the action is prohibited.
Source-stated
Across 1,974 runs, 254 of the 921 runs reaching all required outcomes did not follow the necessary procedure.
Of the 921 runs that reached all required outcomes, 254 (28%) did not follow the necessary procedure.
Source-stated
Claude Opus 5 reached all required outcomes in 81.6% of runs but passed all checks in 52.5%.
Claude Opus 5 reached all required outcomes in 81.6% of its runs, but only 52.5% of its runs passed all checks.
Source-stated
With a verified route, two models' pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%; the authors say route-finding is the main problem for these models.
When we gave two models a verified route, their pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%. For these models, the main problem is to find a route, not to execute it.
Escape Meridian interpretation
A favorable outcome alone may not establish that a patient-advocacy agent respected required permissions and evidence checks; this is an interpretation of the benchmark's distinction between outcome and procedure.
It must reach the correct outcome, and it must follow the necessary procedure.
Source and contributor context
Vasudevan, Shriram, Mehta, Videet, Sweeney, Connor · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
AI agent
An AI system that chooses and carries out actions toward a goal, often using software tools.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: 6feb3bc9-51e0-4a18-8f96-877025eff483.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-07T07:46:11.295Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: fbb2be26796d960eb030afaa56de5391d83d8d147287059c65892a82de3b5a53. Source text hash: 3e74d06c533e049e6d0af5a58a69a0f103a85d428c598307049d0b957f57a2fc.
Revision and source-check history
Version 1 · 2026-11-07T07:46:11.295Z
BABA-Bench’s authors argue patient-advocacy agents must follow rules, not just reach outcomes
The authors of BABA-Bench argue that an helping a patient must achieve the right outcome and follow the required procedure—for example, get permission before sharing records. Their contains 47 interactive worlds based on de-identified patient-advocacy cases reviewed by expert advocates. Agents must uncover information and coordinate with insurers, clinics, suppliers, pharmacies, and community organizations; the benchmark checks each action for permission, supporting evidence, and prohibited conduct. Across 1,974 runs, the authors report that agents sometimes reached all required outcomes by an invalid route: 254 of the 921 successful runs failed procedural checks. Opus 5 reached all required outcomes in 81.6% of its runs, but passed all checks in 52.5%. When two models received a verified route, their pass rates rose substantially. The authors conclude that, for those models, finding a route—not executing one—is the main problem. The abstract says they evaluated 14 . Escape Meridian interpretation: in patient advocacy, a favorable result alone may not show that an agent respected permission and evidence requirements. The benchmark tests those requirements in simulated worlds; the abstract does not establish how its results translate to real patient care. How well do the benchmark’s interactive worlds capture the constraints agents face
Original source
cs.AI · Article
Source details
Original source date: Oct 29, 2026 (2026-10-29).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-07T07:46:11.295Z. First observed: 2026-11-02T05:06:22.143Z. Retrieved: 2026-11-02T11:47:35.456Z.
Limits
The evidence is an abstract; it does not provide detailed methods, case composition, or results beyond those summarized there. The abstract reports results in benchmark worlds and does not establish how they translate to real patient care. The evidence does not specify which two models received verified routes.
Open question
How well do BABA-Bench’s interactive worlds capture the constraints agents would face in actual patient-advocacy cases?
Source-stated
The authors say patient-advocacy agents need both to reach the correct outcome and to follow the necessary procedure, including obtaining permission before sharing records.
It must reach the correct outcome, and it must follow the necessary procedure. For example, the agent must get the patient's permission before it shares the patient's records.
Source-stated
BABA-Bench comprises 47 interactive worlds built from de-identified patient-advocacy cases reviewed by expert advocates; agents must find hidden information and coordinate across organizations.
We introduce BABA-Bench, a benchmark of 47 interactive worlds. We built each world from a de-identified, real patient advocacy case that expert advocates reviewed.
Source-stated
The benchmark checks each action for permission, evidence, and whether the action is prohibited.
The engine checks each action at the time the agent does it. It checks if the agent has permission, if the agent has evidence for the action, and if the action is prohibited.
Source-stated
Across 1,974 runs, 254 of the 921 runs reaching all required outcomes did not follow the necessary procedure.
Of the 921 runs that reached all required outcomes, 254 (28%) did not follow the necessary procedure.
Source-stated
Claude Opus 5 reached all required outcomes in 81.6% of runs but passed all checks in 52.5%.
Claude Opus 5 reached all required outcomes in 81.6% of its runs, but only 52.5% of its runs passed all checks.
Source-stated
With a verified route, two models' pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%; the authors say route-finding is the main problem for these models.
When we gave two models a verified route, their pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%. For these models, the main problem is to find a route, not to execute it.
Escape Meridian interpretation
A favorable outcome alone may not establish that a patient-advocacy agent respected required permissions and evidence checks; this is an interpretation of the benchmark's distinction between outcome and procedure.
It must reach the correct outcome, and it must follow the necessary procedure.
Source and contributor context
Vasudevan, Shriram, Mehta, Videet, Sweeney, Connor · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
AI agent
An AI system that chooses and carries out actions toward a goal, often using software tools.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: 6feb3bc9-51e0-4a18-8f96-877025eff483.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-07T07:46:11.295Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: fbb2be26796d960eb030afaa56de5391d83d8d147287059c65892a82de3b5a53. Source text hash: 3e74d06c533e049e6d0af5a58a69a0f103a85d428c598307049d0b957f57a2fc.
uncertain · 2026-11-07T07:46:11.295Z · anthropic/claude-opus-5.5
Publication: 6feb3bc9-51e0-4a18-8f96-877025eff483
attribution — pass: The reading consistently credits BABA-Bench's authors (Vasudevan, Mehta, Sweeney) and keeps the findings as their reported results. The Escape Meridian interpretation is clearly labeled.
Authors:Shriram Vasudevan, Videet Mehta, Connor Sweeney
summary — uncertain: First, the summary's final paragraph ends with a truncated, unpunctuated fragment ('How well do the benchmark's interactive worlds capture the constraints agents face'). Smallest repair: delete it, since the question field already covers it. Second, the draft says agents 'sometimes' reached outcomes by an invalid route, but the source says 'often', which softens the authors' central finding. Restore 'often'. Third, the sentence about 14 frontier models sits awkwardly after the conclusion; move it next to the 1,974 runs. Otherwise the summary is understandable to a general reader, and the question follows from the evidence.
Agents often reached the correct outcome by an invalid route.
date — pass: originalPublished 2026-10-29 matches the arXiv v1 submission date. The feed date of 2026-11-02 is later and correctly not used.
[Submitted on 29 Oct 2026]
support — pass: All numeric claims match the abstract: 47 worlds, 14 models, 1,974 runs, 254 of 921, 81.6% versus 52.5%, and 29.8% to 91.5% and 44.0% to 97.2%. The interpretation is labeled and conditional. The limitations correctly state that real-world translation is not established and that the two models are not named.
When we gave two models a verified route, their pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%.
media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
An AI system that chooses and carries out actions toward a goal, often using software tools.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceAn AI system that chooses and carries out actions toward a goal, often using software tools.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference