Tal, Liran, Liu, Chris, Fereday, Charlotte, Roth, Sebastian, Clark, Brian · Oct 28, 2026
Liran Tal and coauthors compare SAST with Claude Code on vulnerability finding
Markdown · JSON · Original source
Liran Tal, Chris Liu, Charlotte Fereday, Sebastian Roth and Brian Clark compare deterministic static application security testing (SAST) with AI security agents on VulnBench 2.0, a of attacker-reachable vulnerabilities in open-source web applications. They report that Snyk Code and Code with Opus 5 at xhigh effort found different vulnerabilities, but the difference was too small to distinguish their performance in this benchmark. Together, the tools found 101 of the benchmark’s 108 vulnerabilities; AI configurations also reported at least seven real vulnerabilities absent from the reference set and unreported by Snyk Code. The authors argue that what an AI engine misses depends on its harness as much as on its model. Their examples include review policies that restrict vulnerability classes, whether findings identify the precise line, and how a scanner selects files. Escape Meridian interpretation: the results make a case for evaluating tools together and examining how configuration shapes coverage, rather than treating model choice alone as the answer. The findings come from a specific benchmark and do not establish how the tools perform across software outside it. How well would these differences in coverage and cost hold on vulnerabilities and applications outside the benchmark?
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 28, 2026 (2026-10-28).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-11T04:25:54.163Z. First observed: 2026-10-30T06:42:57.356Z. Retrieved: 2026-10-31T12:47:50.596Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The evaluation covers 108 hand-curated vulnerabilities in 20 open-source web applications, so its findings do not establish performance on software outside that . The authors say the benchmark findings did not distinguish Snyk Code from Code with Opus 5 at xhigh effort; reported performance should not be read as a general ranking. Some cited policy results come from a supplementary run on 19 of 20 applications, as specified in the abstract.
Open question
How well would these differences in coverage and cost hold on vulnerabilities and applications outside the ?
Source-stated
The authors compare deterministic SAST and AI security agents using VulnBench 2.0, a benchmark of 108 attacker-reachable vulnerabilities in 20 open-source web applications.
VulnBench 2.0 is a benchmark of 108 hand-curated, attacker-reachable vulnerabilities in 20 open-source web applications
Source-stated
Snyk Code reached 82.4% macro recall and Claude Code with Opus 5 at xhigh effort reached 79.3%; the authors say their difference was too small to distinguish them on this benchmark.
Snyk Code reached 82.4% macro recall (89 of 108) and Claude Code with Opus 5 at xhigh effort 79.3% (85 of 108); they disagree on 28 vulnerabilities, split 16 to 12, too few to tell them apart.
Source-stated
Together, Snyk Code and Claude Code with Opus 5 at xhigh effort found 101 of the 108 vulnerabilities in the benchmark.
together they reached 101 of 108, the ceiling of all seven comparable configurations combined
Source-stated
AI configurations also reported at least seven real vulnerabilities that were absent from the reference set and were not reported by Snyk Code.
the AI configurations also reported at least seven real vulnerabilities that the reference set did not contain and Snyk Code did not report.
Source-stated
The authors report that results depend on the harness as well as the model, citing changes associated with review policy, line-level localization and file selection.
What an AI engine misses depends on its harness as much as on its model
Escape Meridian interpretation
The benchmark's findings and coverage comparisons do not establish how these tools perform on software outside the tested applications.
VulnBench 2.0 is a benchmark of 108 hand-curated, attacker-reachable vulnerabilities in 20 open-source web applications
Source and contributor context
Tal, Liran, Liu, Chris, Fereday, Charlotte, Roth, Sebastian, Clark, Brian · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: d2ebf4b4-c843-49c4-830d-bb48f7688952.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T04:25:54.163Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 2bbea6830393e1d65d4683296f47221de840b84e6300b969fe0a58e5688ea273. Source text hash: cf7caf35af5c896209a3b3d6e80c3eab10ec1563bc748e5582404d7bbbfc7cf7.
Revision and source-check history
Version 1 · 2026-11-11T04:25:54.163Z
Liran Tal and coauthors compare SAST with Claude Code on vulnerability finding
Liran Tal, Chris Liu, Charlotte Fereday, Sebastian Roth and Brian Clark compare deterministic static application security testing (SAST) with AI security agents on VulnBench 2.0, a of attacker-reachable vulnerabilities in open-source web applications. They report that Snyk Code and Code with Opus 5 at xhigh effort found different vulnerabilities, but the difference was too small to distinguish their performance in this benchmark. Together, the tools found 101 of the benchmark’s 108 vulnerabilities; AI configurations also reported at least seven real vulnerabilities absent from the reference set and unreported by Snyk Code. The authors argue that what an AI engine misses depends on its harness as much as on its model. Their examples include review policies that restrict vulnerability classes, whether findings identify the precise line, and how a scanner selects files. Escape Meridian interpretation: the results make a case for evaluating tools together and examining how configuration shapes coverage, rather than treating model choice alone as the answer. The findings come from a specific benchmark and do not establish how the tools perform across software outside it. How well would these differences in coverage and cost hold on vulnerabilities and applications outside the benchmark?
Original source
cs.AI · Article
Source details
Original source date: Oct 28, 2026 (2026-10-28).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-11T04:25:54.163Z. First observed: 2026-10-30T06:42:57.356Z. Retrieved: 2026-10-31T12:47:50.596Z.
Limits
The evaluation covers 108 hand-curated vulnerabilities in 20 open-source web applications, so its findings do not establish performance on software outside that . The authors say the benchmark findings did not distinguish Snyk Code from Code with Opus 5 at xhigh effort; reported performance should not be read as a general ranking. Some cited policy results come from a supplementary run on 19 of 20 applications, as specified in the abstract.
Open question
How well would these differences in coverage and cost hold on vulnerabilities and applications outside the ?
Source-stated
The authors compare deterministic SAST and AI security agents using VulnBench 2.0, a benchmark of 108 attacker-reachable vulnerabilities in 20 open-source web applications.
VulnBench 2.0 is a benchmark of 108 hand-curated, attacker-reachable vulnerabilities in 20 open-source web applications
Source-stated
Snyk Code reached 82.4% macro recall and Claude Code with Opus 5 at xhigh effort reached 79.3%; the authors say their difference was too small to distinguish them on this benchmark.
Snyk Code reached 82.4% macro recall (89 of 108) and Claude Code with Opus 5 at xhigh effort 79.3% (85 of 108); they disagree on 28 vulnerabilities, split 16 to 12, too few to tell them apart.
Source-stated
Together, Snyk Code and Claude Code with Opus 5 at xhigh effort found 101 of the 108 vulnerabilities in the benchmark.
together they reached 101 of 108, the ceiling of all seven comparable configurations combined
Source-stated
AI configurations also reported at least seven real vulnerabilities that were absent from the reference set and were not reported by Snyk Code.
the AI configurations also reported at least seven real vulnerabilities that the reference set did not contain and Snyk Code did not report.
Source-stated
The authors report that results depend on the harness as well as the model, citing changes associated with review policy, line-level localization and file selection.
What an AI engine misses depends on its harness as much as on its model
Escape Meridian interpretation
The benchmark's findings and coverage comparisons do not establish how these tools perform on software outside the tested applications.
VulnBench 2.0 is a benchmark of 108 hand-curated, attacker-reachable vulnerabilities in 20 open-source web applications
Source and contributor context
Tal, Liran, Liu, Chris, Fereday, Charlotte, Roth, Sebastian, Clark, Brian · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: d2ebf4b4-c843-49c4-830d-bb48f7688952.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T04:25:54.163Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 2bbea6830393e1d65d4683296f47221de840b84e6300b969fe0a58e5688ea273. Source text hash: cf7caf35af5c896209a3b3d6e80c3eab10ec1563bc748e5582404d7bbbfc7cf7.
uncertain · 2026-11-11T04:25:54.163Z · anthropic/claude-opus-5.5
Publication: d2ebf4b4-c843-49c4-830d-bb48f7688952
attribution — pass: The draft names the five listed authors correctly. It keeps the benchmark results and the harness-versus-model argument attributed to them, and it labels the Escape Meridian interpretation separately.
Authors:Liran Tal, Chris Liu, Charlotte Fereday, Sebastian Roth, Brian Clark
summary — uncertain: The draft is mostly faithful, but the central term "harness" is never explained in plain language, even though the authors' main argument turns on it. Readers also get no explanation of "xhigh effort." The examples help, but a newcomer may not see that a harness is the tool wrapper around the model, such as review policy and file selection. The question raises cost, which the summary never introduces. The title singles out Claude Code, while the study evaluated nine configurations. Smallest repair: add a short gloss or tooltip for harness, and either mention that coverage and cost were only loosely related or drop cost from the question.
What an AI engine misses depends on its harness as much as on its model
date — pass: The originalPublished date of 2026-10-28 matches the v1 submission date on arXiv. It is not the later feed date of 2026-10-31.
[Submitted on 28 Oct 2026]
support — pass: The source supports these figures: 82.4% and 79.3% recall, 28 disagreements too few to tell the tools apart, 101 of 108 found combined, and at least seven extra real vulnerabilities. The harness examples are also supported. The interpretation that tools should be evaluated together is labeled and grounded in the authors' statement that neither engine sufficed alone. The caveat about generalizing beyond the benchmark is appropriately conditional.
Neither engine was sufficient alone: together they reached 101 of 108, the ceiling of all seven comparable configurations combined
media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference