When Evaluators of AI Agents Disagree: An Empirical Study of Native Scoring, LLM Judges, and Commit-Boundary Checks on tau2-bench

When Evaluators of AI Agents Disagree: An Empirical Study of Native Scoring, LLM Judges, and Commit-Boundary Checks on tau2-bench

Sungbin Im’s study asks whether benchmark scoring and deployed-service evaluation can be treated as interchangeable. Benchmarks compare an agent’s final state with a gold answer; deployed services may instead use an LLM judge to read the transcript. In 2,160 episodes across tau2-bench’s airline and retail domains, Im compares native scoring, two versions of an LLM judge, and a boundary evaluator: a rule-based check comparing whether a database write was expected with whether one occurred. The judge repeated its verdict 98.2% of the time but agreed with native scoring on 56.2%. In 11 episodes, agents made forbidden writes; the judge passed 10 to 11, while the boundary evaluator caught all 11. Im also reports that verification, correction, and safe-termination gates did not significantly reduce forbidden writes. Traces suggest a possible reason: the gates’ evidence digest omitted order status and customer authentication, leading to valid requests being rejected. The study’s point is that evaluator choice can change the conclusion, and a minimal check can expose failures that judges miss. Its results come from two benchmark domains and human labels on 120 episodes; the abstract does not establish how well the evaluators perform in other settings. The authors release traces, code, prompts, and labels.

Publication: 2c359c90-9c34-48e4-857e-232617108808

attribution — uncertain: The study is correctly credited to its sole listed author, Sungbin Im, in most places. However, the summary's closing sentence ('The authors release traces...') and the fifth claim ('The authors report...') use a plural that implies several authors. Smallest repair: change both to 'Im'. This does not contradict the findings, but it blurs who is making the claims.

Authors:Sungbin Im

summary — uncertain: The research question, method and headline numbers are faithful. Two gaps remain. First, central terms are not explained for a general reader: 'gates' (verification, correction, safe-termination), 'evidence digest', 'forbidden write' and 'episodes'. A reader cannot tell what a gate does or why it would reject valid requests. Second, the summary drops results that carry the paper's force: a stronger judge still passed 6 of the 11 forbidden writes, no judge significantly beat the rule check on human labels, and forbidden writes occurred in 11.1% of episodes without gates. Smallest repair: add one plain-language gloss for gates and digest, plus the stronger-judge result.

A stronger judge still passed 6. The boundary evaluator, with no model call, caught all 11; no judge significantly exceeded it on human labels. Forbidden writes occurred in 11.1% of episodes without gates.

date — pass: originalPublished is 2026-11-10. This matches the arXiv submission date of the v1 release; the feed date of Nov 13 is later and is correctly not used.

[Submitted on 10 Nov 2026]

support — pass: The claims are grounded in the abstract. The generalization limit is labeled as interpretation and does not overreach. The question about adding order status and authentication to the digest follows from the reported candidate cause and is not presented as a finding. The draft keeps the source's hedging ('candidate cause', 'possible reason'). Claim 5 frames the digest omission somewhat narrowly as a cause of valid-request rejections, but it remains consistent with the source wording.

The traces show a candidate cause: the gates' evidence digest dropped the order status and customer authentication, so valid requests were rejected.

media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the original source.