arXiv: Pdb or Not to Be: Comparing Debugging Interfaces With Pass$_{X}$@$k$

arXiv: Pdb or Not to Be: Comparing Debugging Interfaces With Pass$_{X}$@$k$

Authors Ryan Magnuson, Ellie Kitanidis and Ariel Lee compare coding agents that debug through bash commands with agents using an interactive debugger, which exposes a program’s runtime state while it runs. They test both approaches on all 500 SWE-bench Verified instances, keeping the rest of the setup fixed, across two models and three independent runs for each model-and-interface configuration. The authors introduce pass$_X$@$k$, an estimator of the chance that at least one of k agents succeeds, to distinguish a genuinely complementary tool from variation between repeated runs. Although bash and debugger agents solve partly different benchmark instances in individual pairs of runs, the authors say the estimator indicates this difference is largely run-to-run variance. Across both models, they report that two bash rollouts outperform one bash and one debugger rollout. Their argument is that apparent diversity among agent tools can be misleading when repeated sampling is not accounted for: a debugger’s different successes do not, in this evaluation, add as much as another bash attempt. The result concerns these benchmark instances, models and tested setups; the abstract does not establish whether the comparison generalizes beyond them. The authors present pass$_X$@$k$ as a framework for evaluating agent and tool diversity.

Publication: 14d61453-4c56-4197-8b60-809eb31a796f

attribution — pass: The draft correctly attributes the paper and its findings to Ryan Magnuson, Ellie Kitanidis and Ariel Lee. It keeps results framed as what the authors report, for example 'the authors say' and 'they report'.

Authors:Ryan Magnuson, Ellie Kitanidis, Ariel Lee

summary — uncertain: The summary is faithful. However, central terms that a non-specialist needs remain unexplained: SWE-bench Verified (a benchmark of real software bug-fix tasks), rollouts (independent agent attempts) and bash (command-line shell tools). The title and summary also show raw LaTeX notation (pass$_X$@$k$), which a general reader cannot parse. Smallest repair: render the notation as passX@k, add a brief plain-language gloss of SWE-bench Verified and rollout, or add term-library tooltips. Naming the two models (GPT-OSS-120B, Nemotron-3-Ultra-550B) would also help but is optional.

we compare a debugger-only agent against a bash-only agent on all 500 SWE-bench Verified instances, holding the rest of the scaffold fixed.

date — pass: The originalPublished value, 2026-11-05, matches the arXiv v1 submission date. The later feed date of 2026-11-09 is correctly not used.

[Submitted on 5 Nov 2026]

support — pass: The claims match the abstract: the controlled comparison, the estimator's definition, the finding that disjoint successes reflect run-to-run variance, and the result that two bash rollouts beat a bash-plus-debugger pair. The interpretive sentence about apparent tool diversity matches the authors' own conclusion. The generalization question and the limitations are appropriately conditional and add no unsupported benefits or costs.

Our findings highlight the importance of accounting for stochastic variance in evaluating agent and tool diversity

media — pass: The selected media is the verified direct link to the original arXiv abstract page. It is the most direct way to access the source.