{"id":"45d1985c0e00d4f2ba3e9616735c8eff18206dbd9179ae191b56094fb186cc75","title":"BABA-Bench’s authors argue patient-advocacy agents must follow rules, not just reach outcomes","argument":"The authors of BABA-Bench argue that an AI agent helping a patient must achieve the right outcome and follow the required procedure—for example, get permission before sharing records. Their benchmark contains 47 interactive worlds based on de-identified patient-advocacy cases reviewed by expert advocates. Agents must uncover information and coordinate with insurers, clinics, suppliers, pharmacies, and community organizations; the benchmark checks each action for permission, supporting evidence, and prohibited conduct.\n\nAcross 1,974 runs, the authors report that agents sometimes reached all required outcomes by an invalid route: 254 of the 921 successful runs failed procedural checks. Claude Opus 5 reached all required outcomes in 81.6% of its runs, but passed all checks in 52.5%. When two models received a verified route, their pass rates rose substantially. The authors conclude that, for those models, finding a route—not executing one—is the main problem. The abstract says they evaluated 14 frontier models.\n\nEscape Meridian interpretation: in patient advocacy, a favorable result alone may not show that an agent respected permission and evidence requirements. The benchmark tests those requirements in simulated worlds; the abstract does not establish how its results translate to real patient care. How well do the benchmark’s interactive worlds capture the constraints agents face","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence is an arXiv abstract; it does not provide detailed benchmark methods, case composition, or results beyond those summarized there. The abstract reports results in benchmark worlds and does not establish how they translate to real patient care. The evidence does not specify which two models received verified routes.","speaker":{"name":"Vasudevan, Shriram, Mehta, Videet, Sweeney, Connor","roleAtSource":"Article author"},"source":{"title":"BABA-Bench: Evaluating Long-Horizon Patient-Advocacy Agents","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.41076","publishedAt":"2026-10-29","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors say patient-advocacy agents need both to reach the correct outcome and to follow the necessary procedure, including obtaining permission before sharing records.","excerpt":"It must reach the correct outcome, and it must follow the necessary procedure. For example, the agent must get the patient's permission before it shares the patient's records.","url":"https://arxiv.org/abs/2610.41076","relationship":"Source-stated"},{"text":"BABA-Bench comprises 47 interactive worlds built from de-identified patient-advocacy cases reviewed by expert advocates; agents must find hidden information and coordinate across organizations.","excerpt":"We introduce BABA-Bench, a benchmark of 47 interactive worlds. We built each world from a de-identified, real patient advocacy case that expert advocates reviewed.","url":"https://arxiv.org/abs/2610.41076","relationship":"Source-stated"},{"text":"The benchmark checks each action for permission, evidence, and whether the action is prohibited.","excerpt":"The engine checks each action at the time the agent does it. It checks if the agent has permission, if the agent has evidence for the action, and if the action is prohibited.","url":"https://arxiv.org/abs/2610.41076","relationship":"Source-stated"},{"text":"Across 1,974 runs, 254 of the 921 runs reaching all required outcomes did not follow the necessary procedure.","excerpt":"Of the 921 runs that reached all required outcomes, 254 (28%) did not follow the necessary procedure.","url":"https://arxiv.org/abs/2610.41076","relationship":"Source-stated"},{"text":"Claude Opus 5 reached all required outcomes in 81.6% of runs but passed all checks in 52.5%.","excerpt":"Claude Opus 5 reached all required outcomes in 81.6% of its runs, but only 52.5% of its runs passed all checks.","url":"https://arxiv.org/abs/2610.41076","relationship":"Source-stated"},{"text":"With a verified route, two models' pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%; the authors say route-finding is the main problem for these models.","excerpt":"When we gave two models a verified route, their pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%. For these models, the main problem is to find a route, not to execute it.","url":"https://arxiv.org/abs/2610.41076","relationship":"Source-stated"},{"text":"A favorable outcome alone may not establish that a patient-advocacy agent respected required permissions and evidence checks; this is an interpretation of the benchmark's distinction between outcome and procedure.","excerpt":"It must reach the correct outcome, and it must follow the necessary procedure.","url":"https://arxiv.org/abs/2610.41076","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.41076","checkedAt":"2026-11-07T08:50:07.588Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well do BABA-Bench’s interactive worlds capture the constraints agents would face in actual patient-advocacy cases?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"BABA-Bench’s authors argue patient-advocacy agents must follow rules, not just reach outcomes","summary":"The authors of BABA-Bench argue that an AI agent helping a patient must achieve the right outcome and follow the required procedure—for example, get permission before sharing records. Their benchmark contains 47 interactive worlds based on de-identified patient-advocacy cases reviewed by expert advocates. Agents must uncover information and coordinate with insurers, clinics, suppliers, pharmacies, and community organizations; the benchmark checks each action for permission, supporting evidence, and prohibited conduct.\n\nAcross 1,974 runs, the authors report that agents sometimes reached all required outcomes by an invalid route: 254 of the 921 successful runs failed procedural checks. Claude Opus 5 reached all required outcomes in 81.6% of its runs, but passed all checks in 52.5%. When two models received a verified route, their pass rates rose substantially. The authors conclude that, for those models, finding a route—not executing one—is the main problem. The abstract says they evaluated 14 frontier models.\n\nEscape Meridian interpretation: in patient advocacy, a favorable result alone may not show that an agent respected permission and evidence requirements. The benchmark tests those requirements in simulated worlds; the abstract does not establish how its results translate to real patient care. How well do the benchmark’s interactive worlds capture the constraints agents face","question":"How well do BABA-Bench’s interactive worlds capture the constraints agents would face in actual patient-advocacy cases?","claims":[{"text":"The authors say patient-advocacy agents need both to reach the correct outcome and to follow the necessary procedure, including obtaining permission before sharing records.","excerpt":"It must reach the correct outcome, and it must follow the necessary procedure. For example, the agent must get the patient's permission before it shares the patient's records.","support":"source-stated"},{"text":"BABA-Bench comprises 47 interactive worlds built from de-identified patient-advocacy cases reviewed by expert advocates; agents must find hidden information and coordinate across organizations.","excerpt":"We introduce BABA-Bench, a benchmark of 47 interactive worlds. We built each world from a de-identified, real patient advocacy case that expert advocates reviewed.","support":"source-stated"},{"text":"The benchmark checks each action for permission, evidence, and whether the action is prohibited.","excerpt":"The engine checks each action at the time the agent does it. It checks if the agent has permission, if the agent has evidence for the action, and if the action is prohibited.","support":"source-stated"},{"text":"Across 1,974 runs, 254 of the 921 runs reaching all required outcomes did not follow the necessary procedure.","excerpt":"Of the 921 runs that reached all required outcomes, 254 (28%) did not follow the necessary procedure.","support":"source-stated"},{"text":"Claude Opus 5 reached all required outcomes in 81.6% of runs but passed all checks in 52.5%.","excerpt":"Claude Opus 5 reached all required outcomes in 81.6% of its runs, but only 52.5% of its runs passed all checks.","support":"source-stated"},{"text":"With a verified route, two models' pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%; the authors say route-finding is the main problem for these models.","excerpt":"When we gave two models a verified route, their pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%. For these models, the main problem is to find a route, not to execute it.","support":"source-stated"},{"text":"A favorable outcome alone may not establish that a patient-advocacy agent respected required permissions and evidence checks; this is an interpretation of the benchmark's distinction between outcome and procedure.","excerpt":"It must reach the correct outcome, and it must follow the necessary procedure.","support":"interpretation"}],"limitations":["The evidence is an arXiv abstract; it does not provide detailed benchmark methods, case composition, or results beyond those summarized there.","The abstract reports results in benchmark worlds and does not establish how they translate to real patient care.","The evidence does not specify which two models received verified routes."],"mediaId":"8a3d9c31a7be2b116559a1dbc3de79f449509da4ea95212fc9b9eae471b363da","correctionReason":null,"id":"45d1985c0e00d4f2ba3e9616735c8eff18206dbd9179ae191b56094fb186cc75","publicationId":"6feb3bc9-51e0-4a18-8f96-877025eff483","version":1,"sourceUrl":"https://arxiv.org/abs/2610.41076","sourceTitle":"BABA-Bench: Evaluating Long-Horizon Patient-Advocacy Agents","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-29","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-02T05:06:22.143Z","createdAt":"2026-11-07T07:46:11.295Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"8a3d9c31a7be2b116559a1dbc3de79f449509da4ea95212fc9b9eae471b363da","kind":"article","url":"https://arxiv.org/abs/2610.41076","sourceUrl":"https://arxiv.org/abs/2610.41076","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-02T11:47:35.652Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.41076","checkedAt":"2026-11-07T08:50:07.588Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"8a3d9c31a7be2b116559a1dbc3de79f449509da4ea95212fc9b9eae471b363da","confidence":0.81,"provider":"typesafe","model":"jev-1.13.0","inputHash":"02d3f28371665b3c38716314e758f805de18c5edc9d05a8bca0b6c4c6148b8e1","decidedAt":"2026-11-02T11:47:36.750Z"},"provenance":{"textHash":"3e74d06c533e049e6d0af5a58a69a0f103a85d428c598307049d0b957f57a2fc","observationId":"fbb2be26796d960eb030afaa56de5391d83d8d147287059c65892a82de3b5a53","retrievedAt":"2026-11-02T11:47:35.456Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"c60621c4-55d2-47bc-8424-6cba0363f37e"},"reviewer":{"model":"gpt-6-luna","runId":"611e3026-b21b-4530-82c9-e6ecbaefb247"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-07T07:46:11.295Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Vasudevan, Shriram","url":null,"kind":"unknown"},{"name":"Mehta, Videet","url":null,"kind":"unknown"},{"name":"Sweeney, Connor","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Vasudevan, Shriram","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Vasudevan, Shriram","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.41076","locator":"head/meta[name=author|citation_author]"},"assertionId":"1eda7e2f393459c438c5da270f043f87e607d726c5914e1add792426832da9e0:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-07T08:50:07.588Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1eda7e2f393459c438c5da270f043f87e607d726c5914e1add792426832da9e0","sourceUrl":"https://arxiv.org/abs/2610.41076","contentSha256":"7daad1dcd636c0cea97b00eae61500de19aada33e8392c5822f82111e3b3e1db","textHash":"3e74d06c533e049e6d0af5a58a69a0f103a85d428c598307049d0b957f57a2fc","observedAt":"2026-11-07T08:50:07.588Z"}},{"actor":{"name":"Mehta, Videet","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mehta, Videet","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.41076","locator":"head/meta[name=author|citation_author]"},"assertionId":"1eda7e2f393459c438c5da270f043f87e607d726c5914e1add792426832da9e0:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-07T08:50:07.588Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1eda7e2f393459c438c5da270f043f87e607d726c5914e1add792426832da9e0","sourceUrl":"https://arxiv.org/abs/2610.41076","contentSha256":"7daad1dcd636c0cea97b00eae61500de19aada33e8392c5822f82111e3b3e1db","textHash":"3e74d06c533e049e6d0af5a58a69a0f103a85d428c598307049d0b957f57a2fc","observedAt":"2026-11-07T08:50:07.588Z"}},{"actor":{"name":"Sweeney, Connor","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sweeney, Connor","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.41076","locator":"head/meta[name=author|citation_author]"},"assertionId":"1eda7e2f393459c438c5da270f043f87e607d726c5914e1add792426832da9e0:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-07T08:50:07.588Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1eda7e2f393459c438c5da270f043f87e607d726c5914e1add792426832da9e0","sourceUrl":"https://arxiv.org/abs/2610.41076","contentSha256":"7daad1dcd636c0cea97b00eae61500de19aada33e8392c5822f82111e3b3e1db","textHash":"3e74d06c533e049e6d0af5a58a69a0f103a85d428c598307049d0b957f57a2fc","observedAt":"2026-11-07T08:50:07.588Z"}}],"participationCoverage":{"total":3,"returned":3,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.41076","snapshotId":"1eda7e2f393459c438c5da270f043f87e607d726c5914e1add792426832da9e0","contentSha256":"7daad1dcd636c0cea97b00eae61500de19aada33e8392c5822f82111e3b3e1db","textHash":"3e74d06c533e049e6d0af5a58a69a0f103a85d428c598307049d0b957f57a2fc","observedAt":"2026-11-07T08:50:07.588Z","basis":"publisher-page-metadata","sourceLinks":41,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/45d1985c0e00d4f2ba3e9616735c8eff18206dbd9179ae191b56094fb186cc75/source"}},"canonicalUrl":"https://escapemeridian.com/readings/baba-bench-s-authors-argue-patient-advocacy-agents-must-follow-45d1985c","formats":{"html":"/readings/baba-bench-s-authors-argue-patient-advocacy-agents-must-follow-45d1985c","markdown":"/readings/baba-bench-s-authors-argue-patient-advocacy-agents-must-follow-45d1985c.md","json":"/readings/baba-bench-s-authors-argue-patient-advocacy-agents-must-follow-45d1985c.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-ai-agent","name":"AI agent","explanation":"An AI system that chooses and carries out actions toward a goal, often using software tools.","referenceUrl":"https://www.ibm.com/think/topics/ai-agents"},{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-frontier-model","name":"Frontier model","explanation":"One of the most capable AI models available at a given time; the set shifts as models improve.","referenceUrl":"https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"},{"id":"claude","name":"Claude","kind":"model","context":"Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.","referenceUrls":["https://claude.com/product/overview","https://www.anthropic.com/news","https://www.anthropic.com/research/team/economics"],"path":"/entities/claude","json":"/entities/claude.json","markdown":"/entities/claude.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"45d1985c0e00d4f2ba3e9616735c8eff18206dbd9179ae191b56094fb186cc75","revision":"6feb3bc9-51e0-4a18-8f96-877025eff483","revisionKind":"publication-id","title":"BABA-Bench’s authors argue patient-advocacy agents must follow rules, not just reach outcomes","url":"https://escapemeridian.com/readings/baba-bench-s-authors-argue-patient-advocacy-agents-must-follow-45d1985c"},"source":{"type":"source","id":"https://arxiv.org/abs/2610.41076","revision":"3e74d06c533e049e6d0af5a58a69a0f103a85d428c598307049d0b957f57a2fc","revisionKind":"sha256-source-text","title":"BABA-Bench: Evaluating Long-Horizon Patient-Advocacy Agents","url":"https://arxiv.org/abs/2610.41076"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"45d1985c0e00d4f2ba3e9616735c8eff18206dbd9179ae191b56094fb186cc75","title":"BABA-Bench’s authors argue patient-advocacy agents must follow rules, not just reach outcomes","argument":"The authors of BABA-Bench argue that an AI agent helping a patient must achieve the right outcome and follow the required procedure—for example, get permission before sharing records. Their benchmark contains 47 interactive worlds based on de-identified patient-advocacy cases reviewed by expert advocates. Agents must uncover information and coordinate with insurers, clinics, suppliers, pharmacies, and community organizations; the benchmark checks each action for permission, supporting evidence, and prohibited conduct.\n\nAcross 1,974 runs, the authors report that agents sometimes reached all required outcomes by an invalid route: 254 of the 921 successful runs failed procedural checks. Claude Opus 5 reached all required outcomes in 81.6% of its runs, but passed all checks in 52.5%. When two models received a verified route, their pass rates rose substantially. The authors conclude that, for those models, finding a route—not executing one—is the main problem. The abstract says they evaluated 14 frontier models.\n\nEscape Meridian interpretation: in patient advocacy, a favorable result alone may not show that an agent respected permission and evidence requirements. The benchmark tests those requirements in simulated worlds; the abstract does not establish how its results translate to real patient care. How well do the benchmark’s interactive worlds capture the constraints agents face","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence is an arXiv abstract; it does not provide detailed benchmark methods, case composition, or results beyond those summarized there. The abstract reports results in benchmark worlds and does not establish how they translate to real patient care. The evidence does not specify which two models received verified routes.","speaker":{"name":"Vasudevan, Shriram, Mehta, Videet, Sweeney, Connor","roleAtSource":"Article author"},"source":{"title":"BABA-Bench: Evaluating Long-Horizon Patient-Advocacy Agents","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.41076","publishedAt":"2026-10-29","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors say patient-advocacy agents need both to reach the correct outcome and to follow the necessary procedure, including obtaining permission before sharing records.","excerpt":"It must reach the correct outcome, and it must follow the necessary procedure. For example, the agent must get the patient's permission before it shares the patient's records.","url":"https://arxiv.org/abs/2610.41076","relationship":"Source-stated"},{"text":"BABA-Bench comprises 47 interactive worlds built from de-identified patient-advocacy cases reviewed by expert advocates; agents must find hidden information and coordinate across organizations.","excerpt":"We introduce BABA-Bench, a benchmark of 47 interactive worlds. We built each world from a de-identified, real patient advocacy case that expert advocates reviewed.","url":"https://arxiv.org/abs/2610.41076","relationship":"Source-stated"},{"text":"The benchmark checks each action for permission, evidence, and whether the action is prohibited.","excerpt":"The engine checks each action at the time the agent does it. It checks if the agent has permission, if the agent has evidence for the action, and if the action is prohibited.","url":"https://arxiv.org/abs/2610.41076","relationship":"Source-stated"},{"text":"Across 1,974 runs, 254 of the 921 runs reaching all required outcomes did not follow the necessary procedure.","excerpt":"Of the 921 runs that reached all required outcomes, 254 (28%) did not follow the necessary procedure.","url":"https://arxiv.org/abs/2610.41076","relationship":"Source-stated"},{"text":"Claude Opus 5 reached all required outcomes in 81.6% of runs but passed all checks in 52.5%.","excerpt":"Claude Opus 5 reached all required outcomes in 81.6% of its runs, but only 52.5% of its runs passed all checks.","url":"https://arxiv.org/abs/2610.41076","relationship":"Source-stated"},{"text":"With a verified route, two models' pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%; the authors say route-finding is the main problem for these models.","excerpt":"When we gave two models a verified route, their pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%. For these models, the main problem is to find a route, not to execute it.","url":"https://arxiv.org/abs/2610.41076","relationship":"Source-stated"},{"text":"A favorable outcome alone may not establish that a patient-advocacy agent respected required permissions and evidence checks; this is an interpretation of the benchmark's distinction between outcome and procedure.","excerpt":"It must reach the correct outcome, and it must follow the necessary procedure.","url":"https://arxiv.org/abs/2610.41076","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.41076","checkedAt":"2026-11-07T08:50:07.588Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well do BABA-Bench’s interactive worlds capture the constraints agents would face in actual patient-advocacy cases?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"BABA-Bench’s authors argue patient-advocacy agents must follow rules, not just reach outcomes","summary":"The authors of BABA-Bench argue that an AI agent helping a patient must achieve the right outcome and follow the required procedure—for example, get permission before sharing records. Their benchmark contains 47 interactive worlds based on de-identified patient-advocacy cases reviewed by expert advocates. Agents must uncover information and coordinate with insurers, clinics, suppliers, pharmacies, and community organizations; the benchmark checks each action for permission, supporting evidence, and prohibited conduct.\n\nAcross 1,974 runs, the authors report that agents sometimes reached all required outcomes by an invalid route: 254 of the 921 successful runs failed procedural checks. Claude Opus 5 reached all required outcomes in 81.6% of its runs, but passed all checks in 52.5%. When two models received a verified route, their pass rates rose substantially. The authors conclude that, for those models, finding a route—not executing one—is the main problem. The abstract says they evaluated 14 frontier models.\n\nEscape Meridian interpretation: in patient advocacy, a favorable result alone may not show that an agent respected permission and evidence requirements. The benchmark tests those requirements in simulated worlds; the abstract does not establish how its results translate to real patient care. How well do the benchmark’s interactive worlds capture the constraints agents face","question":"How well do BABA-Bench’s interactive worlds capture the constraints agents would face in actual patient-advocacy cases?","claims":[{"text":"The authors say patient-advocacy agents need both to reach the correct outcome and to follow the necessary procedure, including obtaining permission before sharing records.","excerpt":"It must reach the correct outcome, and it must follow the necessary procedure. For example, the agent must get the patient's permission before it shares the patient's records.","support":"source-stated"},{"text":"BABA-Bench comprises 47 interactive worlds built from de-identified patient-advocacy cases reviewed by expert advocates; agents must find hidden information and coordinate across organizations.","excerpt":"We introduce BABA-Bench, a benchmark of 47 interactive worlds. We built each world from a de-identified, real patient advocacy case that expert advocates reviewed.","support":"source-stated"},{"text":"The benchmark checks each action for permission, evidence, and whether the action is prohibited.","excerpt":"The engine checks each action at the time the agent does it. It checks if the agent has permission, if the agent has evidence for the action, and if the action is prohibited.","support":"source-stated"},{"text":"Across 1,974 runs, 254 of the 921 runs reaching all required outcomes did not follow the necessary procedure.","excerpt":"Of the 921 runs that reached all required outcomes, 254 (28%) did not follow the necessary procedure.","support":"source-stated"},{"text":"Claude Opus 5 reached all required outcomes in 81.6% of runs but passed all checks in 52.5%.","excerpt":"Claude Opus 5 reached all required outcomes in 81.6% of its runs, but only 52.5% of its runs passed all checks.","support":"source-stated"},{"text":"With a verified route, two models' pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%; the authors say route-finding is the main problem for these models.","excerpt":"When we gave two models a verified route, their pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%. For these models, the main problem is to find a route, not to execute it.","support":"source-stated"},{"text":"A favorable outcome alone may not establish that a patient-advocacy agent respected required permissions and evidence checks; this is an interpretation of the benchmark's distinction between outcome and procedure.","excerpt":"It must reach the correct outcome, and it must follow the necessary procedure.","support":"interpretation"}],"limitations":["The evidence is an arXiv abstract; it does not provide detailed benchmark methods, case composition, or results beyond those summarized there.","The abstract reports results in benchmark worlds and does not establish how they translate to real patient care.","The evidence does not specify which two models received verified routes."],"mediaId":"8a3d9c31a7be2b116559a1dbc3de79f449509da4ea95212fc9b9eae471b363da","correctionReason":null,"id":"45d1985c0e00d4f2ba3e9616735c8eff18206dbd9179ae191b56094fb186cc75","publicationId":"6feb3bc9-51e0-4a18-8f96-877025eff483","version":1,"sourceUrl":"https://arxiv.org/abs/2610.41076","sourceTitle":"BABA-Bench: Evaluating Long-Horizon Patient-Advocacy Agents","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-29","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-02T05:06:22.143Z","createdAt":"2026-11-07T07:46:11.295Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"8a3d9c31a7be2b116559a1dbc3de79f449509da4ea95212fc9b9eae471b363da","kind":"article","url":"https://arxiv.org/abs/2610.41076","sourceUrl":"https://arxiv.org/abs/2610.41076","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-02T11:47:35.652Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.41076","checkedAt":"2026-11-07T08:50:07.588Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"8a3d9c31a7be2b116559a1dbc3de79f449509da4ea95212fc9b9eae471b363da","confidence":0.81,"provider":"typesafe","model":"jev-1.13.0","inputHash":"02d3f28371665b3c38716314e758f805de18c5edc9d05a8bca0b6c4c6148b8e1","decidedAt":"2026-11-02T11:47:36.750Z"},"provenance":{"textHash":"3e74d06c533e049e6d0af5a58a69a0f103a85d428c598307049d0b957f57a2fc","observationId":"fbb2be26796d960eb030afaa56de5391d83d8d147287059c65892a82de3b5a53","retrievedAt":"2026-11-02T11:47:35.456Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"c60621c4-55d2-47bc-8424-6cba0363f37e"},"reviewer":{"model":"gpt-6-luna","runId":"611e3026-b21b-4530-82c9-e6ecbaefb247"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-07T07:46:11.295Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Vasudevan, Shriram","url":null,"kind":"unknown"},{"name":"Mehta, Videet","url":null,"kind":"unknown"},{"name":"Sweeney, Connor","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Vasudevan, Shriram","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Vasudevan, Shriram","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.41076","locator":"head/meta[name=author|citation_author]"},"assertionId":"1eda7e2f393459c438c5da270f043f87e607d726c5914e1add792426832da9e0:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-07T08:50:07.588Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1eda7e2f393459c438c5da270f043f87e607d726c5914e1add792426832da9e0","sourceUrl":"https://arxiv.org/abs/2610.41076","contentSha256":"7daad1dcd636c0cea97b00eae61500de19aada33e8392c5822f82111e3b3e1db","textHash":"3e74d06c533e049e6d0af5a58a69a0f103a85d428c598307049d0b957f57a2fc","observedAt":"2026-11-07T08:50:07.588Z"}},{"actor":{"name":"Mehta, Videet","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mehta, Videet","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.41076","locator":"head/meta[name=author|citation_author]"},"assertionId":"1eda7e2f393459c438c5da270f043f87e607d726c5914e1add792426832da9e0:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-07T08:50:07.588Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1eda7e2f393459c438c5da270f043f87e607d726c5914e1add792426832da9e0","sourceUrl":"https://arxiv.org/abs/2610.41076","contentSha256":"7daad1dcd636c0cea97b00eae61500de19aada33e8392c5822f82111e3b3e1db","textHash":"3e74d06c533e049e6d0af5a58a69a0f103a85d428c598307049d0b957f57a2fc","observedAt":"2026-11-07T08:50:07.588Z"}},{"actor":{"name":"Sweeney, Connor","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sweeney, Connor","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.41076","locator":"head/meta[name=author|citation_author]"},"assertionId":"1eda7e2f393459c438c5da270f043f87e607d726c5914e1add792426832da9e0:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-07T08:50:07.588Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1eda7e2f393459c438c5da270f043f87e607d726c5914e1add792426832da9e0","sourceUrl":"https://arxiv.org/abs/2610.41076","contentSha256":"7daad1dcd636c0cea97b00eae61500de19aada33e8392c5822f82111e3b3e1db","textHash":"3e74d06c533e049e6d0af5a58a69a0f103a85d428c598307049d0b957f57a2fc","observedAt":"2026-11-07T08:50:07.588Z"}}],"participationCoverage":{"total":3,"returned":3,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.41076","snapshotId":"1eda7e2f393459c438c5da270f043f87e607d726c5914e1add792426832da9e0","contentSha256":"7daad1dcd636c0cea97b00eae61500de19aada33e8392c5822f82111e3b3e1db","textHash":"3e74d06c533e049e6d0af5a58a69a0f103a85d428c598307049d0b957f57a2fc","observedAt":"2026-11-07T08:50:07.588Z","basis":"publisher-page-metadata","sourceLinks":41,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/45d1985c0e00d4f2ba3e9616735c8eff18206dbd9179ae191b56094fb186cc75/source"}}}],"audits":[{"publicationId":"6feb3bc9-51e0-4a18-8f96-877025eff483","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading consistently credits BABA-Bench's authors (Vasudevan, Mehta, Sweeney) and keeps the findings as their reported results. The Escape Meridian interpretation is clearly labeled.","excerpt":"Authors:Shriram Vasudevan, Videet Mehta, Connor Sweeney"},{"dimension":"summary","status":"uncertain","explanation":"First, the summary's final paragraph ends with a truncated, unpunctuated fragment ('How well do the benchmark's interactive worlds capture the constraints agents face'). Smallest repair: delete it, since the question field already covers it. Second, the draft says agents 'sometimes' reached outcomes by an invalid route, but the source says 'often', which softens the authors' central finding. Restore 'often'. Third, the sentence about 14 frontier models sits awkwardly after the conclusion; move it next to the 1,974 runs. Otherwise the summary is understandable to a general reader, and the question follows from the evidence.","excerpt":"Agents often reached the correct outcome by an invalid route."},{"dimension":"date","status":"pass","explanation":"originalPublished 2026-10-29 matches the arXiv v1 submission date. The feed date of 2026-11-02 is later and correctly not used.","excerpt":"[Submitted on 29 Oct 2026]"},{"dimension":"support","status":"pass","explanation":"All numeric claims match the abstract: 47 worlds, 14 models, 1,974 runs, 254 of 921, 81.6% versus 52.5%, and 29.8% to 91.5% and 44.0% to 97.2%. The interpretation is labeled and conditional. The limitations correctly state that real-world translation is not established and that the two models are not named.","excerpt":"When we gave two models a verified route, their pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.","excerpt":null}],"createdAt":"2026-11-07T07:46:11.295Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}