{"id":"062c189481d743c500f5fb150982cfbb757eed79c268debe00034f325129d588","title":"arXiv cs.AI: PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix","argument":"In their paper, Marco Paina, Mark Niklas Müller, Niels Mündler-Sasahara and Martin Vechev introduce PROBE, a benchmark built from bugs fixed across 20 GitHub repositories over 18 months. It compares coding agents’ findings with this record of bugs that maintainers repaired. The authors report that agents had recall below 21% and precision below 9% against the reference set. Recall measures how many reference bugs an agent finds; precision measures how many of its findings match that set. Stronger models raised recall from 8% to 17%, but did not improve precision. The authors also report that evaluators judged 85% of findings valid, and that agents’ bug distributions differed substantially from the bugs maintainers fixed. Escape Meridian’s interpretation is that findings judged valid do not necessarily match the bugs maintainers repair, though the benchmark’s reference set is limited to bugs that were fixed. The paper presents PROBE as a resource for model users and developers. How well would its findings and comparisons hold across repositories or bugs beyond those in the benchmark?","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reference set is limited to bugs that maintainers fixed in the included repositories; the abstract does not establish how the results generalize beyond that set. The abstract does not specify the evaluation details or define how evaluators judged findings valid.","speaker":{"name":"Paina, Marco, Müller, Mark Niklas, Mündler-Sasahara, Niels, Vechev, Martin","roleAtSource":"Article author"},"source":{"title":"PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.35987","publishedAt":"2026-10-26","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce PROBE, a benchmark of bugs fixed across 20 GitHub repositories over 18 months.","excerpt":"a benchmark consisting of all 2,457 bugs fixed over the course of 18 months across 20 diverse GitHub repositories","url":"https://arxiv.org/abs/2610.35987","relationship":"Source-stated"},{"text":"Against the benchmark’s reference set, evaluated coding agents had recall below 21% and precision below 9%.","excerpt":"they all achieve a recall of below $21\\%$ and a precision of less than $9\\%$ with respect to the reference set.","url":"https://arxiv.org/abs/2610.35987","relationship":"Source-stated"},{"text":"The authors report that stronger models raised recall from 8% to 17%, without improving precision.","excerpt":"stronger models increase recall substantially from $8\\%$ to $17\\%$, they do not improve precision.","url":"https://arxiv.org/abs/2610.35987","relationship":"Source-stated"},{"text":"Evaluators judged 85% of findings valid, while the distribution of bugs found differed substantially from the distribution of bugs maintainers fixed.","excerpt":"LLM and human evaluators still judge $85\\%$ of findings valid. Crucially, we find that found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers.","url":"https://arxiv.org/abs/2610.35987","relationship":"Source-stated"},{"text":"The findings suggest that being judged valid does not necessarily mean a bug matches the kinds of bugs maintainers repair.","excerpt":"LLM and human evaluators still judge $85\\%$ of findings valid. Crucially, we find that found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers.","url":"https://arxiv.org/abs/2610.35987","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.35987","checkedAt":"2026-11-05T17:34:09.053Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well would PROBE’s findings and comparisons hold across repositories or bugs beyond those in the benchmark?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix","summary":"In their paper, Marco Paina, Mark Niklas Müller, Niels Mündler-Sasahara and Martin Vechev introduce PROBE, a benchmark built from bugs fixed across 20 GitHub repositories over 18 months. It compares coding agents’ findings with this record of bugs that maintainers repaired. The authors report that agents had recall below 21% and precision below 9% against the reference set. Recall measures how many reference bugs an agent finds; precision measures how many of its findings match that set. Stronger models raised recall from 8% to 17%, but did not improve precision. The authors also report that evaluators judged 85% of findings valid, and that agents’ bug distributions differed substantially from the bugs maintainers fixed. Escape Meridian’s interpretation is that findings judged valid do not necessarily match the bugs maintainers repair, though the benchmark’s reference set is limited to bugs that were fixed. The paper presents PROBE as a resource for model users and developers. How well would its findings and comparisons hold across repositories or bugs beyond those in the benchmark?","question":"How well would PROBE’s findings and comparisons hold across repositories or bugs beyond those in the benchmark?","claims":[{"text":"The authors introduce PROBE, a benchmark of bugs fixed across 20 GitHub repositories over 18 months.","excerpt":"a benchmark consisting of all 2,457 bugs fixed over the course of 18 months across 20 diverse GitHub repositories","support":"source-stated"},{"text":"Against the benchmark’s reference set, evaluated coding agents had recall below 21% and precision below 9%.","excerpt":"they all achieve a recall of below $21\\%$ and a precision of less than $9\\%$ with respect to the reference set.","support":"source-stated"},{"text":"The authors report that stronger models raised recall from 8% to 17%, without improving precision.","excerpt":"stronger models increase recall substantially from $8\\%$ to $17\\%$, they do not improve precision.","support":"source-stated"},{"text":"Evaluators judged 85% of findings valid, while the distribution of bugs found differed substantially from the distribution of bugs maintainers fixed.","excerpt":"LLM and human evaluators still judge $85\\%$ of findings valid. Crucially, we find that found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers.","support":"source-stated"},{"text":"The findings suggest that being judged valid does not necessarily mean a bug matches the kinds of bugs maintainers repair.","excerpt":"LLM and human evaluators still judge $85\\%$ of findings valid. Crucially, we find that found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers.","support":"interpretation"}],"limitations":["The reference set is limited to bugs that maintainers fixed in the included repositories; the abstract does not establish how the results generalize beyond that set.","The abstract does not specify the evaluation details or define how evaluators judged findings valid."],"mediaId":"f444f34121adaeaae7bb7e84326847391cde36f87c61c8983a2c048c76181e63","correctionReason":null,"id":"062c189481d743c500f5fb150982cfbb757eed79c268debe00034f325129d588","publicationId":"b27f0614-77bf-41f9-8739-2e34885b4d0a","version":1,"sourceUrl":"https://arxiv.org/abs/2610.35987","sourceTitle":"PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-26","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-28T05:30:33.483Z","createdAt":"2026-11-05T17:02:18.012Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"f444f34121adaeaae7bb7e84326847391cde36f87c61c8983a2c048c76181e63","kind":"article","url":"https://arxiv.org/abs/2610.35987","sourceUrl":"https://arxiv.org/abs/2610.35987","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-29T16:18:53.154Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.35987","checkedAt":"2026-11-05T17:34:09.053Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"f444f34121adaeaae7bb7e84326847391cde36f87c61c8983a2c048c76181e63","confidence":0.82,"provider":"typesafe","model":"jev-1.13.0","inputHash":"7f7fc979da782614e1172553e72f8de47b5b44fedc98af115b6c7ff682029292","decidedAt":"2026-10-29T16:18:54.894Z"},"provenance":{"textHash":"e21c6de666e410271a308e78feb81b99400a2aed353eb19bbd18690e836ea11b","observationId":"e4e783f58e7a3360780770558c17a23303c6cc3cba66d3d7955c1fa687b81772","retrievedAt":"2026-10-29T16:18:52.867Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"f82f5b66-39b5-4cf2-8d61-8623d75cd769"},"reviewer":{"model":"gpt-6-luna","runId":"408d8ace-50c5-4df8-80bd-87f534a10c61"}},"audit":{"status":"passed","reviewedAt":"2026-11-05T17:02:18.012Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Paina, Marco","url":null,"kind":"unknown"},{"name":"Müller, Mark Niklas","url":null,"kind":"unknown"},{"name":"Mündler-Sasahara, Niels","url":null,"kind":"unknown"},{"name":"Vechev, Martin","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Paina, Marco","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Paina, Marco","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35987","locator":"head/meta[name=author|citation_author]"},"assertionId":"2cbcedc959536adfa45d76faaa44f6fa70ea1f707f6cf8c614319b23a0b70822:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-29T16:18:52.867Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2cbcedc959536adfa45d76faaa44f6fa70ea1f707f6cf8c614319b23a0b70822","sourceUrl":"https://arxiv.org/abs/2610.35987","contentSha256":"af0538ac568ee1e8aa0034016911dd6c7887fa9ec738f7da5948dd7723b47342","textHash":"e21c6de666e410271a308e78feb81b99400a2aed353eb19bbd18690e836ea11b","observedAt":"2026-10-29T16:18:52.867Z"}},{"actor":{"name":"Müller, Mark Niklas","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Müller, Mark Niklas","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35987","locator":"head/meta[name=author|citation_author]"},"assertionId":"2cbcedc959536adfa45d76faaa44f6fa70ea1f707f6cf8c614319b23a0b70822:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-29T16:18:52.867Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2cbcedc959536adfa45d76faaa44f6fa70ea1f707f6cf8c614319b23a0b70822","sourceUrl":"https://arxiv.org/abs/2610.35987","contentSha256":"af0538ac568ee1e8aa0034016911dd6c7887fa9ec738f7da5948dd7723b47342","textHash":"e21c6de666e410271a308e78feb81b99400a2aed353eb19bbd18690e836ea11b","observedAt":"2026-10-29T16:18:52.867Z"}},{"actor":{"name":"Mündler-Sasahara, Niels","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mündler-Sasahara, Niels","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35987","locator":"head/meta[name=author|citation_author]"},"assertionId":"2cbcedc959536adfa45d76faaa44f6fa70ea1f707f6cf8c614319b23a0b70822:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-29T16:18:52.867Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2cbcedc959536adfa45d76faaa44f6fa70ea1f707f6cf8c614319b23a0b70822","sourceUrl":"https://arxiv.org/abs/2610.35987","contentSha256":"af0538ac568ee1e8aa0034016911dd6c7887fa9ec738f7da5948dd7723b47342","textHash":"e21c6de666e410271a308e78feb81b99400a2aed353eb19bbd18690e836ea11b","observedAt":"2026-10-29T16:18:52.867Z"}},{"actor":{"name":"Vechev, Martin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Vechev, Martin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35987","locator":"head/meta[name=author|citation_author]"},"assertionId":"2cbcedc959536adfa45d76faaa44f6fa70ea1f707f6cf8c614319b23a0b70822:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-29T16:18:52.867Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2cbcedc959536adfa45d76faaa44f6fa70ea1f707f6cf8c614319b23a0b70822","sourceUrl":"https://arxiv.org/abs/2610.35987","contentSha256":"af0538ac568ee1e8aa0034016911dd6c7887fa9ec738f7da5948dd7723b47342","textHash":"e21c6de666e410271a308e78feb81b99400a2aed353eb19bbd18690e836ea11b","observedAt":"2026-10-29T16:18:52.867Z"}}],"participationCoverage":{"total":4,"returned":4,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.35987","snapshotId":"2cbcedc959536adfa45d76faaa44f6fa70ea1f707f6cf8c614319b23a0b70822","contentSha256":"af0538ac568ee1e8aa0034016911dd6c7887fa9ec738f7da5948dd7723b47342","textHash":"e21c6de666e410271a308e78feb81b99400a2aed353eb19bbd18690e836ea11b","observedAt":"2026-10-29T16:18:52.867Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/062c189481d743c500f5fb150982cfbb757eed79c268debe00034f325129d588/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-probe-frontier-coding-agents-find-different-bugs-062c1894","formats":{"html":"/readings/arxiv-cs-ai-probe-frontier-coding-agents-find-different-bugs-062c1894","markdown":"/readings/arxiv-cs-ai-probe-frontier-coding-agents-find-different-bugs-062c1894.md","json":"/readings/arxiv-cs-ai-probe-frontier-coding-agents-find-different-bugs-062c1894.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"062c189481d743c500f5fb150982cfbb757eed79c268debe00034f325129d588","revision":"b27f0614-77bf-41f9-8739-2e34885b4d0a","revisionKind":"publication-id","title":"arXiv cs.AI: PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix","url":"https://escapemeridian.com/readings/arxiv-cs-ai-probe-frontier-coding-agents-find-different-bugs-062c1894"},"source":{"type":"source","id":"https://arxiv.org/abs/2610.35987","revision":"e21c6de666e410271a308e78feb81b99400a2aed353eb19bbd18690e836ea11b","revisionKind":"sha256-source-text","title":"PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix","url":"https://arxiv.org/abs/2610.35987"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"062c189481d743c500f5fb150982cfbb757eed79c268debe00034f325129d588","title":"arXiv cs.AI: PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix","argument":"In their paper, Marco Paina, Mark Niklas Müller, Niels Mündler-Sasahara and Martin Vechev introduce PROBE, a benchmark built from bugs fixed across 20 GitHub repositories over 18 months. It compares coding agents’ findings with this record of bugs that maintainers repaired. The authors report that agents had recall below 21% and precision below 9% against the reference set. Recall measures how many reference bugs an agent finds; precision measures how many of its findings match that set. Stronger models raised recall from 8% to 17%, but did not improve precision. The authors also report that evaluators judged 85% of findings valid, and that agents’ bug distributions differed substantially from the bugs maintainers fixed. Escape Meridian’s interpretation is that findings judged valid do not necessarily match the bugs maintainers repair, though the benchmark’s reference set is limited to bugs that were fixed. The paper presents PROBE as a resource for model users and developers. How well would its findings and comparisons hold across repositories or bugs beyond those in the benchmark?","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reference set is limited to bugs that maintainers fixed in the included repositories; the abstract does not establish how the results generalize beyond that set. The abstract does not specify the evaluation details or define how evaluators judged findings valid.","speaker":{"name":"Paina, Marco, Müller, Mark Niklas, Mündler-Sasahara, Niels, Vechev, Martin","roleAtSource":"Article author"},"source":{"title":"PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.35987","publishedAt":"2026-10-26","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce PROBE, a benchmark of bugs fixed across 20 GitHub repositories over 18 months.","excerpt":"a benchmark consisting of all 2,457 bugs fixed over the course of 18 months across 20 diverse GitHub repositories","url":"https://arxiv.org/abs/2610.35987","relationship":"Source-stated"},{"text":"Against the benchmark’s reference set, evaluated coding agents had recall below 21% and precision below 9%.","excerpt":"they all achieve a recall of below $21\\%$ and a precision of less than $9\\%$ with respect to the reference set.","url":"https://arxiv.org/abs/2610.35987","relationship":"Source-stated"},{"text":"The authors report that stronger models raised recall from 8% to 17%, without improving precision.","excerpt":"stronger models increase recall substantially from $8\\%$ to $17\\%$, they do not improve precision.","url":"https://arxiv.org/abs/2610.35987","relationship":"Source-stated"},{"text":"Evaluators judged 85% of findings valid, while the distribution of bugs found differed substantially from the distribution of bugs maintainers fixed.","excerpt":"LLM and human evaluators still judge $85\\%$ of findings valid. Crucially, we find that found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers.","url":"https://arxiv.org/abs/2610.35987","relationship":"Source-stated"},{"text":"The findings suggest that being judged valid does not necessarily mean a bug matches the kinds of bugs maintainers repair.","excerpt":"LLM and human evaluators still judge $85\\%$ of findings valid. Crucially, we find that found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers.","url":"https://arxiv.org/abs/2610.35987","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.35987","checkedAt":"2026-11-05T17:34:09.053Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well would PROBE’s findings and comparisons hold across repositories or bugs beyond those in the benchmark?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix","summary":"In their paper, Marco Paina, Mark Niklas Müller, Niels Mündler-Sasahara and Martin Vechev introduce PROBE, a benchmark built from bugs fixed across 20 GitHub repositories over 18 months. It compares coding agents’ findings with this record of bugs that maintainers repaired. The authors report that agents had recall below 21% and precision below 9% against the reference set. Recall measures how many reference bugs an agent finds; precision measures how many of its findings match that set. Stronger models raised recall from 8% to 17%, but did not improve precision. The authors also report that evaluators judged 85% of findings valid, and that agents’ bug distributions differed substantially from the bugs maintainers fixed. Escape Meridian’s interpretation is that findings judged valid do not necessarily match the bugs maintainers repair, though the benchmark’s reference set is limited to bugs that were fixed. The paper presents PROBE as a resource for model users and developers. How well would its findings and comparisons hold across repositories or bugs beyond those in the benchmark?","question":"How well would PROBE’s findings and comparisons hold across repositories or bugs beyond those in the benchmark?","claims":[{"text":"The authors introduce PROBE, a benchmark of bugs fixed across 20 GitHub repositories over 18 months.","excerpt":"a benchmark consisting of all 2,457 bugs fixed over the course of 18 months across 20 diverse GitHub repositories","support":"source-stated"},{"text":"Against the benchmark’s reference set, evaluated coding agents had recall below 21% and precision below 9%.","excerpt":"they all achieve a recall of below $21\\%$ and a precision of less than $9\\%$ with respect to the reference set.","support":"source-stated"},{"text":"The authors report that stronger models raised recall from 8% to 17%, without improving precision.","excerpt":"stronger models increase recall substantially from $8\\%$ to $17\\%$, they do not improve precision.","support":"source-stated"},{"text":"Evaluators judged 85% of findings valid, while the distribution of bugs found differed substantially from the distribution of bugs maintainers fixed.","excerpt":"LLM and human evaluators still judge $85\\%$ of findings valid. Crucially, we find that found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers.","support":"source-stated"},{"text":"The findings suggest that being judged valid does not necessarily mean a bug matches the kinds of bugs maintainers repair.","excerpt":"LLM and human evaluators still judge $85\\%$ of findings valid. Crucially, we find that found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers.","support":"interpretation"}],"limitations":["The reference set is limited to bugs that maintainers fixed in the included repositories; the abstract does not establish how the results generalize beyond that set.","The abstract does not specify the evaluation details or define how evaluators judged findings valid."],"mediaId":"f444f34121adaeaae7bb7e84326847391cde36f87c61c8983a2c048c76181e63","correctionReason":null,"id":"062c189481d743c500f5fb150982cfbb757eed79c268debe00034f325129d588","publicationId":"b27f0614-77bf-41f9-8739-2e34885b4d0a","version":1,"sourceUrl":"https://arxiv.org/abs/2610.35987","sourceTitle":"PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-26","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-28T05:30:33.483Z","createdAt":"2026-11-05T17:02:18.012Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"f444f34121adaeaae7bb7e84326847391cde36f87c61c8983a2c048c76181e63","kind":"article","url":"https://arxiv.org/abs/2610.35987","sourceUrl":"https://arxiv.org/abs/2610.35987","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-29T16:18:53.154Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.35987","checkedAt":"2026-11-05T17:34:09.053Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"f444f34121adaeaae7bb7e84326847391cde36f87c61c8983a2c048c76181e63","confidence":0.82,"provider":"typesafe","model":"jev-1.13.0","inputHash":"7f7fc979da782614e1172553e72f8de47b5b44fedc98af115b6c7ff682029292","decidedAt":"2026-10-29T16:18:54.894Z"},"provenance":{"textHash":"e21c6de666e410271a308e78feb81b99400a2aed353eb19bbd18690e836ea11b","observationId":"e4e783f58e7a3360780770558c17a23303c6cc3cba66d3d7955c1fa687b81772","retrievedAt":"2026-10-29T16:18:52.867Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"f82f5b66-39b5-4cf2-8d61-8623d75cd769"},"reviewer":{"model":"gpt-6-luna","runId":"408d8ace-50c5-4df8-80bd-87f534a10c61"}},"audit":{"status":"passed","reviewedAt":"2026-11-05T17:02:18.012Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Paina, Marco","url":null,"kind":"unknown"},{"name":"Müller, Mark Niklas","url":null,"kind":"unknown"},{"name":"Mündler-Sasahara, Niels","url":null,"kind":"unknown"},{"name":"Vechev, Martin","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Paina, Marco","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Paina, Marco","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35987","locator":"head/meta[name=author|citation_author]"},"assertionId":"2cbcedc959536adfa45d76faaa44f6fa70ea1f707f6cf8c614319b23a0b70822:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-29T16:18:52.867Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2cbcedc959536adfa45d76faaa44f6fa70ea1f707f6cf8c614319b23a0b70822","sourceUrl":"https://arxiv.org/abs/2610.35987","contentSha256":"af0538ac568ee1e8aa0034016911dd6c7887fa9ec738f7da5948dd7723b47342","textHash":"e21c6de666e410271a308e78feb81b99400a2aed353eb19bbd18690e836ea11b","observedAt":"2026-10-29T16:18:52.867Z"}},{"actor":{"name":"Müller, Mark Niklas","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Müller, Mark Niklas","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35987","locator":"head/meta[name=author|citation_author]"},"assertionId":"2cbcedc959536adfa45d76faaa44f6fa70ea1f707f6cf8c614319b23a0b70822:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-29T16:18:52.867Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2cbcedc959536adfa45d76faaa44f6fa70ea1f707f6cf8c614319b23a0b70822","sourceUrl":"https://arxiv.org/abs/2610.35987","contentSha256":"af0538ac568ee1e8aa0034016911dd6c7887fa9ec738f7da5948dd7723b47342","textHash":"e21c6de666e410271a308e78feb81b99400a2aed353eb19bbd18690e836ea11b","observedAt":"2026-10-29T16:18:52.867Z"}},{"actor":{"name":"Mündler-Sasahara, Niels","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mündler-Sasahara, Niels","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35987","locator":"head/meta[name=author|citation_author]"},"assertionId":"2cbcedc959536adfa45d76faaa44f6fa70ea1f707f6cf8c614319b23a0b70822:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-29T16:18:52.867Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2cbcedc959536adfa45d76faaa44f6fa70ea1f707f6cf8c614319b23a0b70822","sourceUrl":"https://arxiv.org/abs/2610.35987","contentSha256":"af0538ac568ee1e8aa0034016911dd6c7887fa9ec738f7da5948dd7723b47342","textHash":"e21c6de666e410271a308e78feb81b99400a2aed353eb19bbd18690e836ea11b","observedAt":"2026-10-29T16:18:52.867Z"}},{"actor":{"name":"Vechev, Martin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Vechev, Martin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35987","locator":"head/meta[name=author|citation_author]"},"assertionId":"2cbcedc959536adfa45d76faaa44f6fa70ea1f707f6cf8c614319b23a0b70822:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-29T16:18:52.867Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2cbcedc959536adfa45d76faaa44f6fa70ea1f707f6cf8c614319b23a0b70822","sourceUrl":"https://arxiv.org/abs/2610.35987","contentSha256":"af0538ac568ee1e8aa0034016911dd6c7887fa9ec738f7da5948dd7723b47342","textHash":"e21c6de666e410271a308e78feb81b99400a2aed353eb19bbd18690e836ea11b","observedAt":"2026-10-29T16:18:52.867Z"}}],"participationCoverage":{"total":4,"returned":4,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.35987","snapshotId":"2cbcedc959536adfa45d76faaa44f6fa70ea1f707f6cf8c614319b23a0b70822","contentSha256":"af0538ac568ee1e8aa0034016911dd6c7887fa9ec738f7da5948dd7723b47342","textHash":"e21c6de666e410271a308e78feb81b99400a2aed353eb19bbd18690e836ea11b","observedAt":"2026-10-29T16:18:52.867Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/062c189481d743c500f5fb150982cfbb757eed79c268debe00034f325129d588/source"}}}],"audits":[{"publicationId":"b27f0614-77bf-41f9-8739-2e34885b4d0a","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The draft correctly names the paper's four authors and keeps the results attributed to them as their reported findings. The publisher label arXiv cs.AI matches the metadata; the paper is primarily cs.SE but is cross-listed in cs.AI.","excerpt":"Authors:Marco Paina, Mark Niklas Müller, Niels Mündler-Sasahara, Martin Vechev"},{"dimension":"summary","status":"pass","explanation":"The summary states the development, the key numbers and the central divergence finding clearly. It defines recall and precision in plain language and labels the Escape Meridian inference. Optional fidelity repairs: the draft says only 'evaluators' where the source says 'LLM and human evaluators', and it drops the companion point that found-bug distributions are similar across agents. Adding both would preserve the full 'Crucially' finding. The question is a genuine open issue.","excerpt":"found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers"},{"dimension":"date","status":"pass","explanation":"originalPublished 2026-10-26 matches the arXiv v1 submission date. The feed date of Oct 29 is correctly not used.","excerpt":"[Submitted on 26 Oct 2026]"},{"dimension":"support","status":"pass","explanation":"The numeric claims match the abstract verbatim. The interpretation that findings judged valid need not match maintainer-fixed bugs is labeled, supported by the 85% validity versus low precision and the distribution difference, and hedged by the limitation that the reference set covers only fixed bugs. This is consistent with the source's 'partial reference set'. No benefits or costs are invented.","excerpt":"together with a method for matching candidate findings to this partial reference set"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.","excerpt":null}],"createdAt":"2026-11-05T17:02:18.012Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}