Paina, Marco, Müller, Mark Niklas, Mündler-Sasahara, Niels, Vechev, Martin · Oct 26, 2026
arXiv cs.AI: PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix
Markdown · JSON · Original source
In their paper, Marco Paina, Mark Niklas Müller, Niels Mündler-Sasahara and Martin Vechev introduce PROBE, a built from bugs fixed across 20 GitHub repositories over 18 months. It compares coding agents’ findings with this record of bugs that maintainers repaired. The authors report that agents had recall below 21% and precision below 9% against the reference set. Recall measures how many reference bugs an agent finds; precision measures how many of its findings match that set. Stronger models raised recall from 8% to 17%, but did not improve precision. The authors also report that evaluators judged 85% of findings valid, and that agents’ bug distributions differed substantially from the bugs maintainers fixed. Escape Meridian’s interpretation is that findings judged valid do not necessarily match the bugs maintainers repair, though the benchmark’s reference set is limited to bugs that were fixed. The paper presents PROBE as a resource for model users and developers. How well would its findings and comparisons hold across repositories or bugs beyond those in the benchmark?
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 26, 2026 (2026-10-26).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-05T17:02:18.012Z. First observed: 2026-10-28T05:30:33.483Z. Retrieved: 2026-10-29T16:18:52.867Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The reference set is limited to bugs that maintainers fixed in the included repositories; the abstract does not establish how the results generalize beyond that set. The abstract does not specify the evaluation details or define how evaluators judged findings valid.
Open question
How well would PROBE’s findings and comparisons hold across repositories or bugs beyond those in the ?
Source-stated
The authors introduce PROBE, a benchmark of bugs fixed across 20 GitHub repositories over 18 months.
a benchmark consisting of all 2,457 bugs fixed over the course of 18 months across 20 diverse GitHub repositories
Source-stated
Against the benchmark’s reference set, evaluated coding agents had recall below 21% and precision below 9%.
they all achieve a recall of below $21\%$ and a precision of less than $9\%$ with respect to the reference set.
Source-stated
The authors report that stronger models raised recall from 8% to 17%, without improving precision.
stronger models increase recall substantially from $8\%$ to $17\%$, they do not improve precision.
Source-stated
Evaluators judged 85% of findings valid, while the distribution of bugs found differed substantially from the distribution of bugs maintainers fixed.
LLM and human evaluators still judge $85\%$ of findings valid. Crucially, we find that found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers.
Escape Meridian interpretation
The findings suggest that being judged valid does not necessarily mean a bug matches the kinds of bugs maintainers repair.
LLM and human evaluators still judge $85\%$ of findings valid. Crucially, we find that found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers.
Source and contributor context
Paina, Marco, Müller, Mark Niklas, Mündler-Sasahara, Niels, Vechev, Martin · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: b27f0614-77bf-41f9-8739-2e34885b4d0a.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-05T17:02:18.012Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: e4e783f58e7a3360780770558c17a23303c6cc3cba66d3d7955c1fa687b81772. Source text hash: e21c6de666e410271a308e78feb81b99400a2aed353eb19bbd18690e836ea11b.
Revision and source-check history
Version 1 · 2026-11-05T17:02:18.012Z
arXiv cs.AI: PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix
In their paper, Marco Paina, Mark Niklas Müller, Niels Mündler-Sasahara and Martin Vechev introduce PROBE, a built from bugs fixed across 20 GitHub repositories over 18 months. It compares coding agents’ findings with this record of bugs that maintainers repaired. The authors report that agents had recall below 21% and precision below 9% against the reference set. Recall measures how many reference bugs an agent finds; precision measures how many of its findings match that set. Stronger models raised recall from 8% to 17%, but did not improve precision. The authors also report that evaluators judged 85% of findings valid, and that agents’ bug distributions differed substantially from the bugs maintainers fixed. Escape Meridian’s interpretation is that findings judged valid do not necessarily match the bugs maintainers repair, though the benchmark’s reference set is limited to bugs that were fixed. The paper presents PROBE as a resource for model users and developers. How well would its findings and comparisons hold across repositories or bugs beyond those in the benchmark?
Original source
cs.AI · Article
Source details
Original source date: Oct 26, 2026 (2026-10-26).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-05T17:02:18.012Z. First observed: 2026-10-28T05:30:33.483Z. Retrieved: 2026-10-29T16:18:52.867Z.
Limits
The reference set is limited to bugs that maintainers fixed in the included repositories; the abstract does not establish how the results generalize beyond that set. The abstract does not specify the evaluation details or define how evaluators judged findings valid.
Open question
How well would PROBE’s findings and comparisons hold across repositories or bugs beyond those in the ?
Source-stated
The authors introduce PROBE, a benchmark of bugs fixed across 20 GitHub repositories over 18 months.
a benchmark consisting of all 2,457 bugs fixed over the course of 18 months across 20 diverse GitHub repositories
Source-stated
Against the benchmark’s reference set, evaluated coding agents had recall below 21% and precision below 9%.
they all achieve a recall of below $21\%$ and a precision of less than $9\%$ with respect to the reference set.
Source-stated
The authors report that stronger models raised recall from 8% to 17%, without improving precision.
stronger models increase recall substantially from $8\%$ to $17\%$, they do not improve precision.
Source-stated
Evaluators judged 85% of findings valid, while the distribution of bugs found differed substantially from the distribution of bugs maintainers fixed.
LLM and human evaluators still judge $85\%$ of findings valid. Crucially, we find that found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers.
Escape Meridian interpretation
The findings suggest that being judged valid does not necessarily mean a bug matches the kinds of bugs maintainers repair.
LLM and human evaluators still judge $85\%$ of findings valid. Crucially, we find that found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers.
Source and contributor context
Paina, Marco, Müller, Mark Niklas, Mündler-Sasahara, Niels, Vechev, Martin · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: b27f0614-77bf-41f9-8739-2e34885b4d0a.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-05T17:02:18.012Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: e4e783f58e7a3360780770558c17a23303c6cc3cba66d3d7955c1fa687b81772. Source text hash: e21c6de666e410271a308e78feb81b99400a2aed353eb19bbd18690e836ea11b.
passed · 2026-11-05T17:02:18.012Z · anthropic/claude-opus-5.5
Publication: b27f0614-77bf-41f9-8739-2e34885b4d0a
attribution — pass: The draft correctly names the paper's four authors and keeps the results attributed to them as their reported findings. The publisher label arXiv cs.AI matches the metadata; the paper is primarily cs.SE but is cross-listed in cs.AI.
Authors:Marco Paina, Mark Niklas Müller, Niels Mündler-Sasahara, Martin Vechev
summary — pass: The summary states the development, the key numbers and the central divergence finding clearly. It defines recall and precision in plain language and labels the Escape Meridian inference. Optional fidelity repairs: the draft says only 'evaluators' where the source says 'LLM and human evaluators', and it drops the companion point that found-bug distributions are similar across agents. Adding both would preserve the full 'Crucially' finding. The question is a genuine open issue.
found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers
date — pass: originalPublished 2026-10-26 matches the arXiv v1 submission date. The feed date of Oct 29 is correctly not used.
[Submitted on 26 Oct 2026]
support — pass: The numeric claims match the abstract verbatim. The interpretation that findings judged valid need not match maintainer-fixed bugs is labeled, supported by the 85% validity versus low precision and the distribution difference, and hedged by the limitation that the reference set covers only fixed bugs. This is consistent with the source's 'partial reference set'. No benefits or costs are invented.
together with a method for matching candidate findings to this partial reference set
media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference