Barazandeh, Babak, Majumdar, Subhabrata, Michailidis, George · Jul 19, 2026
arXiv cs.AI: OTAP evaluates agent trajectories against valid solution graphs
Markdown · JSON · Original source
Babak Barazandeh, Subhabrata Majumdar and George Michailidis propose OTAP, a way to evaluate an agent’s planning and execution by comparing its execution graph—the steps and dependencies in a run—with graphs of valid solutions. Unlike a binary success flag or exact matching, OTAP gives a normalized discrepancy score; its unbalanced marginals allow missing or steps without forcing them to match. The authors report that, on controlled perturbations and three public , OTAP separated valid from invalid trajectories where semantics-only metrics scored below chance. In a blinded study of 236 trajectories, it had the strongest agreement among six compared metrics. These results are encouraging, not a universal measure of agent quality: where a formal verifier exists, strict surface metrics predicted validity better. The authors place OTAP instead in open-ended domains without a verifier. They also report sensitivity to dependency extraction, and say granularity invariance remains an open problem that soft coupling only partly mitigates.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Jul 19, 2026 (2026-07-19).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T00:10:28.495Z. First observed: 2026-11-05T05:00:27.136Z. Retrieved: 2026-11-09T15:46:30.118Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The evidence is the abstract; it does not provide the full paper’s experimental details or results. The reported and blinded-study findings are limited to the settings described in the abstract; the abstract does not establish performance across other tasks or domains. The authors report sensitivity to dependency extraction and say granularity invariance remains an open problem.
Open question
How does OTAP perform when trajectories differ in granularity, given that the authors say granularity invariance remains an open problem?
Source-stated
The authors define OTAP as a normalized discrepancy score comparing an agent’s execution graph with a set of valid solution graphs.
We reframe trajectory evaluation as a distance between the agent's execution graph and a set of valid solution graphs
Source-stated
The authors say OTAP's unbalanced marginals allow missing or hallucinated steps without forcing a match, while granularity invariance remains unresolved.
Its unbalanced marginals handle missing or hallucinated steps without forcing a match; granularity invariance remains an open problem
Source-stated
On controlled perturbations and three public benchmarks, the authors report that OTAP separated valid from invalid trajectories in a regime where semantics-only metrics scored below chance.
On controlled perturbations and three public benchmarks, OTAP separates valid from invalid trajectories in a regime where semantics-only metrics score below chance.
Source-stated
The authors report that strict surface metrics predicted validity better than OTAP where a formal verifier exists, and describe OTAP as suited to open-ended domains without one.
Where a formal verifier exists, strict surface metrics predict validity better than OTAP does, which places OTAP in open-ended domains where no verifier is available.
Source-stated
The authors report sensitivity to dependency extraction in controlled edge-noise experiments.
while controlled edge-noise experiments reveal sensitivity to dependency extraction.
Source and contributor context
Barazandeh, Babak, Majumdar, Subhabrata, Michailidis, George · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Hallucination
When an AI system states something false or made up as if it were true.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: a0d92c09-36e7-44c4-804e-f897468c47a8.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T00:10:28.495Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 14dd983a4b4116ed4aca90a8c96c75036e31e8852bf0c6487adc86227a18a117. Source text hash: 4f3c4d686bf5a23f3c4b0e55227a6bd6d2ec4144e3459d2abfe66835151b5986.
Revision and source-check history
Version 1 · 2026-11-10T00:10:28.495Z
arXiv cs.AI: OTAP evaluates agent trajectories against valid solution graphs
Babak Barazandeh, Subhabrata Majumdar and George Michailidis propose OTAP, a way to evaluate an agent’s planning and execution by comparing its execution graph—the steps and dependencies in a run—with graphs of valid solutions. Unlike a binary success flag or exact matching, OTAP gives a normalized discrepancy score; its unbalanced marginals allow missing or steps without forcing them to match. The authors report that, on controlled perturbations and three public , OTAP separated valid from invalid trajectories where semantics-only metrics scored below chance. In a blinded study of 236 trajectories, it had the strongest agreement among six compared metrics. These results are encouraging, not a universal measure of agent quality: where a formal verifier exists, strict surface metrics predicted validity better. The authors place OTAP instead in open-ended domains without a verifier. They also report sensitivity to dependency extraction, and say granularity invariance remains an open problem that soft coupling only partly mitigates.
Original source
cs.AI · Article
Source details
Original source date: Jul 19, 2026 (2026-07-19).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T00:10:28.495Z. First observed: 2026-11-05T05:00:27.136Z. Retrieved: 2026-11-09T15:46:30.118Z.
Limits
The evidence is the abstract; it does not provide the full paper’s experimental details or results. The reported and blinded-study findings are limited to the settings described in the abstract; the abstract does not establish performance across other tasks or domains. The authors report sensitivity to dependency extraction and say granularity invariance remains an open problem.
Open question
How does OTAP perform when trajectories differ in granularity, given that the authors say granularity invariance remains an open problem?
Source-stated
The authors define OTAP as a normalized discrepancy score comparing an agent’s execution graph with a set of valid solution graphs.
We reframe trajectory evaluation as a distance between the agent's execution graph and a set of valid solution graphs
Source-stated
The authors say OTAP's unbalanced marginals allow missing or hallucinated steps without forcing a match, while granularity invariance remains unresolved.
Its unbalanced marginals handle missing or hallucinated steps without forcing a match; granularity invariance remains an open problem
Source-stated
On controlled perturbations and three public benchmarks, the authors report that OTAP separated valid from invalid trajectories in a regime where semantics-only metrics scored below chance.
On controlled perturbations and three public benchmarks, OTAP separates valid from invalid trajectories in a regime where semantics-only metrics score below chance.
Source-stated
The authors report that strict surface metrics predicted validity better than OTAP where a formal verifier exists, and describe OTAP as suited to open-ended domains without one.
Where a formal verifier exists, strict surface metrics predict validity better than OTAP does, which places OTAP in open-ended domains where no verifier is available.
Source-stated
The authors report sensitivity to dependency extraction in controlled edge-noise experiments.
while controlled edge-noise experiments reveal sensitivity to dependency extraction.
Source and contributor context
Barazandeh, Babak, Majumdar, Subhabrata, Michailidis, George · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Hallucination
When an AI system states something false or made up as if it were true.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: a0d92c09-36e7-44c4-804e-f897468c47a8.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T00:10:28.495Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 14dd983a4b4116ed4aca90a8c96c75036e31e8852bf0c6487adc86227a18a117. Source text hash: 4f3c4d686bf5a23f3c4b0e55227a6bd6d2ec4144e3459d2abfe66835151b5986.
uncertain · 2026-11-10T00:10:28.495Z · anthropic/claude-opus-5.5
Publication: a0d92c09-36e7-44c4-804e-f897468c47a8
attribution — pass: The draft correctly names Babak Barazandeh, Subhabrata Majumdar and George Michailidis as the authors of the arXiv paper. It also attributes the method and the results to them as reported findings rather than established facts.
Authors:Babak Barazandeh, Subhabrata Majumdar, George Michailidis
summary — uncertain: The summary is faithful but loses two key points. It omits the authors' motivation: success flags cannot tell sound runs from lucky ones, and exact matching penalizes valid reordered plans. It also omits the caveat that structure-aware baselines score higher clean-graph AUROC, while OTAP mainly shows the smallest decline under edge noise. Without that caveat, readers may overrate OTAP. Smallest repairs: add one motivation sentence and the AUROC trade-off. Also give a plain gloss of 'agent' and 'granularity invariance', which the question depends on.
A success flag cannot distinguish sound solutions from lucky ones, nor explain why runs fail. Exact matching penalizes valid plans reordered or decomposed differently.
date — pass: The originalPublished date of 2026-07-19 matches the v1 submission date. It is not the later v3 revision of 3 Nov 2026 or the feed date of 5 Nov 2026.
Submitted on 19 Jul 2026 (v1), last revised 3 Nov 2026 (this version, v3)
support — pass: Each claim matches a verbatim excerpt and is framed as the authors' report. 'Encouraging' echoes the source's own word. The verifier caveat, the dependency-extraction sensitivity and the open granularity problem are all preserved. The limitations do not invent benefits or costs, and the question follows from a gap the source itself states.
Its performance is encouraging across these settings, while controlled edge-noise experiments reveal sensitivity to dependency extraction.
media — pass: The selected media is the verified direct link to the arXiv abstract page. It is the original source, and no better source media was found.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
When an AI system states something false or made up as if it were true.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceWhen an AI system states something false or made up as if it were true.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference