{"id":"f93051a53e268749548a10fdbc8c1ab437b1e84e3f93f9c380cd86bc2d1f822f","title":"Liran Tal and coauthors compare SAST with Claude Code on vulnerability finding","argument":"Liran Tal, Chris Liu, Charlotte Fereday, Sebastian Roth and Brian Clark compare deterministic static application security testing (SAST) with AI security agents on VulnBench 2.0, a benchmark of attacker-reachable vulnerabilities in open-source web applications. They report that Snyk Code and Claude Code with Opus 5 at xhigh effort found different vulnerabilities, but the difference was too small to distinguish their performance in this benchmark. Together, the tools found 101 of the benchmark’s 108 vulnerabilities; AI configurations also reported at least seven real vulnerabilities absent from the reference set and unreported by Snyk Code. The authors argue that what an AI engine misses depends on its harness as much as on its model. Their examples include review policies that restrict vulnerability classes, whether findings identify the precise line, and how a scanner selects files. Escape Meridian interpretation: the results make a case for evaluating tools together and examining how configuration shapes coverage, rather than treating model choice alone as the answer. The findings come from a specific benchmark and do not establish how the tools perform across software outside it. How well would these differences in coverage and cost hold on vulnerabilities and applications outside the benchmark?","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evaluation covers 108 hand-curated vulnerabilities in 20 open-source web applications, so its findings do not establish performance on software outside that benchmark. The authors say the benchmark findings did not distinguish Snyk Code from Claude Code with Opus 5 at xhigh effort; reported performance should not be read as a general ranking. Some cited policy results come from a supplementary run on 19 of 20 applications, as specified in the abstract.","speaker":{"name":"Tal, Liran, Liu, Chris, Fereday, Charlotte, Roth, Sebastian, Clark, Brian","roleAtSource":"Article author"},"source":{"title":"How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.38617","publishedAt":"2026-10-28","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors compare deterministic SAST and AI security agents using VulnBench 2.0, a benchmark of 108 attacker-reachable vulnerabilities in 20 open-source web applications.","excerpt":"VulnBench 2.0 is a benchmark of 108 hand-curated, attacker-reachable vulnerabilities in 20 open-source web applications","url":"https://arxiv.org/abs/2610.38617","relationship":"Source-stated"},{"text":"Snyk Code reached 82.4% macro recall and Claude Code with Opus 5 at xhigh effort reached 79.3%; the authors say their difference was too small to distinguish them on this benchmark.","excerpt":"Snyk Code reached 82.4% macro recall (89 of 108) and Claude Code with Opus 5 at xhigh effort 79.3% (85 of 108); they disagree on 28 vulnerabilities, split 16 to 12, too few to tell them apart.","url":"https://arxiv.org/abs/2610.38617","relationship":"Source-stated"},{"text":"Together, Snyk Code and Claude Code with Opus 5 at xhigh effort found 101 of the 108 vulnerabilities in the benchmark.","excerpt":"together they reached 101 of 108, the ceiling of all seven comparable configurations combined","url":"https://arxiv.org/abs/2610.38617","relationship":"Source-stated"},{"text":"AI configurations also reported at least seven real vulnerabilities that were absent from the reference set and were not reported by Snyk Code.","excerpt":"the AI configurations also reported at least seven real vulnerabilities that the reference set did not contain and Snyk Code did not report.","url":"https://arxiv.org/abs/2610.38617","relationship":"Source-stated"},{"text":"The authors report that results depend on the harness as well as the model, citing changes associated with review policy, line-level localization and file selection.","excerpt":"What an AI engine misses depends on its harness as much as on its model","url":"https://arxiv.org/abs/2610.38617","relationship":"Source-stated"},{"text":"The benchmark's findings and coverage comparisons do not establish how these tools perform on software outside the tested applications.","excerpt":"VulnBench 2.0 is a benchmark of 108 hand-curated, attacker-reachable vulnerabilities in 20 open-source web applications","url":"https://arxiv.org/abs/2610.38617","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.38617","checkedAt":"2026-11-11T04:56:08.979Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well would these differences in coverage and cost hold on vulnerabilities and applications outside the benchmark?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Liran Tal and coauthors compare SAST with Claude Code on vulnerability finding","summary":"Liran Tal, Chris Liu, Charlotte Fereday, Sebastian Roth and Brian Clark compare deterministic static application security testing (SAST) with AI security agents on VulnBench 2.0, a benchmark of attacker-reachable vulnerabilities in open-source web applications. They report that Snyk Code and Claude Code with Opus 5 at xhigh effort found different vulnerabilities, but the difference was too small to distinguish their performance in this benchmark. Together, the tools found 101 of the benchmark’s 108 vulnerabilities; AI configurations also reported at least seven real vulnerabilities absent from the reference set and unreported by Snyk Code. The authors argue that what an AI engine misses depends on its harness as much as on its model. Their examples include review policies that restrict vulnerability classes, whether findings identify the precise line, and how a scanner selects files. Escape Meridian interpretation: the results make a case for evaluating tools together and examining how configuration shapes coverage, rather than treating model choice alone as the answer. The findings come from a specific benchmark and do not establish how the tools perform across software outside it. How well would these differences in coverage and cost hold on vulnerabilities and applications outside the benchmark?","question":"How well would these differences in coverage and cost hold on vulnerabilities and applications outside the benchmark?","claims":[{"text":"The authors compare deterministic SAST and AI security agents using VulnBench 2.0, a benchmark of 108 attacker-reachable vulnerabilities in 20 open-source web applications.","excerpt":"VulnBench 2.0 is a benchmark of 108 hand-curated, attacker-reachable vulnerabilities in 20 open-source web applications","support":"source-stated"},{"text":"Snyk Code reached 82.4% macro recall and Claude Code with Opus 5 at xhigh effort reached 79.3%; the authors say their difference was too small to distinguish them on this benchmark.","excerpt":"Snyk Code reached 82.4% macro recall (89 of 108) and Claude Code with Opus 5 at xhigh effort 79.3% (85 of 108); they disagree on 28 vulnerabilities, split 16 to 12, too few to tell them apart.","support":"source-stated"},{"text":"Together, Snyk Code and Claude Code with Opus 5 at xhigh effort found 101 of the 108 vulnerabilities in the benchmark.","excerpt":"together they reached 101 of 108, the ceiling of all seven comparable configurations combined","support":"source-stated"},{"text":"AI configurations also reported at least seven real vulnerabilities that were absent from the reference set and were not reported by Snyk Code.","excerpt":"the AI configurations also reported at least seven real vulnerabilities that the reference set did not contain and Snyk Code did not report.","support":"source-stated"},{"text":"The authors report that results depend on the harness as well as the model, citing changes associated with review policy, line-level localization and file selection.","excerpt":"What an AI engine misses depends on its harness as much as on its model","support":"source-stated"},{"text":"The benchmark's findings and coverage comparisons do not establish how these tools perform on software outside the tested applications.","excerpt":"VulnBench 2.0 is a benchmark of 108 hand-curated, attacker-reachable vulnerabilities in 20 open-source web applications","support":"interpretation"}],"limitations":["The evaluation covers 108 hand-curated vulnerabilities in 20 open-source web applications, so its findings do not establish performance on software outside that benchmark.","The authors say the benchmark findings did not distinguish Snyk Code from Claude Code with Opus 5 at xhigh effort; reported performance should not be read as a general ranking.","Some cited policy results come from a supplementary run on 19 of 20 applications, as specified in the abstract."],"mediaId":"cc8eb88055d274e42ef06f39ab1d0fd3cf2f9d59e08eaaa2b6d2ea3d4947bcb3","correctionReason":null,"id":"f93051a53e268749548a10fdbc8c1ab437b1e84e3f93f9c380cd86bc2d1f822f","publicationId":"d2ebf4b4-c843-49c4-830d-bb48f7688952","version":1,"sourceUrl":"https://arxiv.org/abs/2610.38617","sourceTitle":"How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-28","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-30T06:42:57.356Z","createdAt":"2026-11-11T04:25:54.163Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"cc8eb88055d274e42ef06f39ab1d0fd3cf2f9d59e08eaaa2b6d2ea3d4947bcb3","kind":"article","url":"https://arxiv.org/abs/2610.38617","sourceUrl":"https://arxiv.org/abs/2610.38617","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-31T12:47:55.077Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.38617","checkedAt":"2026-11-11T04:56:08.979Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"cc8eb88055d274e42ef06f39ab1d0fd3cf2f9d59e08eaaa2b6d2ea3d4947bcb3","confidence":0.8899999999999999,"provider":"typesafe","model":"jev-1.13.0","inputHash":"215844944e0cf3c1daec8de802cf1505cc5054998e22380de268d4b0cf02b5a4","decidedAt":"2026-10-31T12:47:59.053Z"},"provenance":{"textHash":"cf7caf35af5c896209a3b3d6e80c3eab10ec1563bc748e5582404d7bbbfc7cf7","observationId":"2bbea6830393e1d65d4683296f47221de840b84e6300b969fe0a58e5688ea273","retrievedAt":"2026-10-31T12:47:50.596Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"ba103904-9fbd-4f8f-813f-edc0592aa012"},"reviewer":{"model":"gpt-6-luna","runId":"7d23ef82-0afe-4c2a-840c-cf89ba3c6ba8"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-11T04:25:54.163Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Tal, Liran","url":null,"kind":"unknown"},{"name":"Liu, Chris","url":null,"kind":"unknown"},{"name":"Fereday, Charlotte","url":null,"kind":"unknown"},{"name":"Roth, Sebastian","url":null,"kind":"unknown"},{"name":"Clark, Brian","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Tal, Liran","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Tal, Liran","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38617","locator":"head/meta[name=author|citation_author]"},"assertionId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T04:56:08.979Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8","sourceUrl":"https://arxiv.org/abs/2610.38617","contentSha256":"85089978de728c91a9bf83b0c026bb4ca8c26e785543f87adbb2b818750d7cca","textHash":"cf7caf35af5c896209a3b3d6e80c3eab10ec1563bc748e5582404d7bbbfc7cf7","observedAt":"2026-11-11T04:56:08.979Z"}},{"actor":{"name":"Liu, Chris","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liu, Chris","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38617","locator":"head/meta[name=author|citation_author]"},"assertionId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T04:56:08.979Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8","sourceUrl":"https://arxiv.org/abs/2610.38617","contentSha256":"85089978de728c91a9bf83b0c026bb4ca8c26e785543f87adbb2b818750d7cca","textHash":"cf7caf35af5c896209a3b3d6e80c3eab10ec1563bc748e5582404d7bbbfc7cf7","observedAt":"2026-11-11T04:56:08.979Z"}},{"actor":{"name":"Fereday, Charlotte","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Fereday, Charlotte","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38617","locator":"head/meta[name=author|citation_author]"},"assertionId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T04:56:08.979Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8","sourceUrl":"https://arxiv.org/abs/2610.38617","contentSha256":"85089978de728c91a9bf83b0c026bb4ca8c26e785543f87adbb2b818750d7cca","textHash":"cf7caf35af5c896209a3b3d6e80c3eab10ec1563bc748e5582404d7bbbfc7cf7","observedAt":"2026-11-11T04:56:08.979Z"}},{"actor":{"name":"Roth, Sebastian","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Roth, Sebastian","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38617","locator":"head/meta[name=author|citation_author]"},"assertionId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T04:56:08.979Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8","sourceUrl":"https://arxiv.org/abs/2610.38617","contentSha256":"85089978de728c91a9bf83b0c026bb4ca8c26e785543f87adbb2b818750d7cca","textHash":"cf7caf35af5c896209a3b3d6e80c3eab10ec1563bc748e5582404d7bbbfc7cf7","observedAt":"2026-11-11T04:56:08.979Z"}},{"actor":{"name":"Clark, Brian","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Clark, Brian","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38617","locator":"head/meta[name=author|citation_author]"},"assertionId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T04:56:08.979Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8","sourceUrl":"https://arxiv.org/abs/2610.38617","contentSha256":"85089978de728c91a9bf83b0c026bb4ca8c26e785543f87adbb2b818750d7cca","textHash":"cf7caf35af5c896209a3b3d6e80c3eab10ec1563bc748e5582404d7bbbfc7cf7","observedAt":"2026-11-11T04:56:08.979Z"}}],"participationCoverage":{"total":5,"returned":5,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.38617","snapshotId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8","contentSha256":"85089978de728c91a9bf83b0c026bb4ca8c26e785543f87adbb2b818750d7cca","textHash":"cf7caf35af5c896209a3b3d6e80c3eab10ec1563bc748e5582404d7bbbfc7cf7","observedAt":"2026-11-11T04:56:08.979Z","basis":"publisher-page-metadata","sourceLinks":45,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/f93051a53e268749548a10fdbc8c1ab437b1e84e3f93f9c380cd86bc2d1f822f/source"}},"canonicalUrl":"https://escapemeridian.com/readings/liran-tal-and-coauthors-compare-sast-with-claude-code-on-f93051a5","formats":{"html":"/readings/liran-tal-and-coauthors-compare-sast-with-claude-code-on-f93051a5","markdown":"/readings/liran-tal-and-coauthors-compare-sast-with-claude-code-on-f93051a5.md","json":"/readings/liran-tal-and-coauthors-compare-sast-with-claude-code-on-f93051a5.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"}],"entities":[{"id":"claude","name":"Claude","kind":"model","context":"Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.","referenceUrls":["https://claude.com/product/overview","https://www.anthropic.com/news","https://www.anthropic.com/research/team/economics"],"path":"/entities/claude","json":"/entities/claude.json","markdown":"/entities/claude.md"},{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"f93051a53e268749548a10fdbc8c1ab437b1e84e3f93f9c380cd86bc2d1f822f","revision":"d2ebf4b4-c843-49c4-830d-bb48f7688952","revisionKind":"publication-id","title":"Liran Tal and coauthors compare SAST with Claude Code on vulnerability finding","url":"https://escapemeridian.com/readings/liran-tal-and-coauthors-compare-sast-with-claude-code-on-f93051a5"},"source":{"type":"source","id":"https://arxiv.org/abs/2610.38617","revision":"cf7caf35af5c896209a3b3d6e80c3eab10ec1563bc748e5582404d7bbbfc7cf7","revisionKind":"sha256-source-text","title":"How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0","url":"https://arxiv.org/abs/2610.38617"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"f93051a53e268749548a10fdbc8c1ab437b1e84e3f93f9c380cd86bc2d1f822f","title":"Liran Tal and coauthors compare SAST with Claude Code on vulnerability finding","argument":"Liran Tal, Chris Liu, Charlotte Fereday, Sebastian Roth and Brian Clark compare deterministic static application security testing (SAST) with AI security agents on VulnBench 2.0, a benchmark of attacker-reachable vulnerabilities in open-source web applications. They report that Snyk Code and Claude Code with Opus 5 at xhigh effort found different vulnerabilities, but the difference was too small to distinguish their performance in this benchmark. Together, the tools found 101 of the benchmark’s 108 vulnerabilities; AI configurations also reported at least seven real vulnerabilities absent from the reference set and unreported by Snyk Code. The authors argue that what an AI engine misses depends on its harness as much as on its model. Their examples include review policies that restrict vulnerability classes, whether findings identify the precise line, and how a scanner selects files. Escape Meridian interpretation: the results make a case for evaluating tools together and examining how configuration shapes coverage, rather than treating model choice alone as the answer. The findings come from a specific benchmark and do not establish how the tools perform across software outside it. How well would these differences in coverage and cost hold on vulnerabilities and applications outside the benchmark?","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evaluation covers 108 hand-curated vulnerabilities in 20 open-source web applications, so its findings do not establish performance on software outside that benchmark. The authors say the benchmark findings did not distinguish Snyk Code from Claude Code with Opus 5 at xhigh effort; reported performance should not be read as a general ranking. Some cited policy results come from a supplementary run on 19 of 20 applications, as specified in the abstract.","speaker":{"name":"Tal, Liran, Liu, Chris, Fereday, Charlotte, Roth, Sebastian, Clark, Brian","roleAtSource":"Article author"},"source":{"title":"How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.38617","publishedAt":"2026-10-28","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors compare deterministic SAST and AI security agents using VulnBench 2.0, a benchmark of 108 attacker-reachable vulnerabilities in 20 open-source web applications.","excerpt":"VulnBench 2.0 is a benchmark of 108 hand-curated, attacker-reachable vulnerabilities in 20 open-source web applications","url":"https://arxiv.org/abs/2610.38617","relationship":"Source-stated"},{"text":"Snyk Code reached 82.4% macro recall and Claude Code with Opus 5 at xhigh effort reached 79.3%; the authors say their difference was too small to distinguish them on this benchmark.","excerpt":"Snyk Code reached 82.4% macro recall (89 of 108) and Claude Code with Opus 5 at xhigh effort 79.3% (85 of 108); they disagree on 28 vulnerabilities, split 16 to 12, too few to tell them apart.","url":"https://arxiv.org/abs/2610.38617","relationship":"Source-stated"},{"text":"Together, Snyk Code and Claude Code with Opus 5 at xhigh effort found 101 of the 108 vulnerabilities in the benchmark.","excerpt":"together they reached 101 of 108, the ceiling of all seven comparable configurations combined","url":"https://arxiv.org/abs/2610.38617","relationship":"Source-stated"},{"text":"AI configurations also reported at least seven real vulnerabilities that were absent from the reference set and were not reported by Snyk Code.","excerpt":"the AI configurations also reported at least seven real vulnerabilities that the reference set did not contain and Snyk Code did not report.","url":"https://arxiv.org/abs/2610.38617","relationship":"Source-stated"},{"text":"The authors report that results depend on the harness as well as the model, citing changes associated with review policy, line-level localization and file selection.","excerpt":"What an AI engine misses depends on its harness as much as on its model","url":"https://arxiv.org/abs/2610.38617","relationship":"Source-stated"},{"text":"The benchmark's findings and coverage comparisons do not establish how these tools perform on software outside the tested applications.","excerpt":"VulnBench 2.0 is a benchmark of 108 hand-curated, attacker-reachable vulnerabilities in 20 open-source web applications","url":"https://arxiv.org/abs/2610.38617","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.38617","checkedAt":"2026-11-11T04:56:08.979Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well would these differences in coverage and cost hold on vulnerabilities and applications outside the benchmark?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Liran Tal and coauthors compare SAST with Claude Code on vulnerability finding","summary":"Liran Tal, Chris Liu, Charlotte Fereday, Sebastian Roth and Brian Clark compare deterministic static application security testing (SAST) with AI security agents on VulnBench 2.0, a benchmark of attacker-reachable vulnerabilities in open-source web applications. They report that Snyk Code and Claude Code with Opus 5 at xhigh effort found different vulnerabilities, but the difference was too small to distinguish their performance in this benchmark. Together, the tools found 101 of the benchmark’s 108 vulnerabilities; AI configurations also reported at least seven real vulnerabilities absent from the reference set and unreported by Snyk Code. The authors argue that what an AI engine misses depends on its harness as much as on its model. Their examples include review policies that restrict vulnerability classes, whether findings identify the precise line, and how a scanner selects files. Escape Meridian interpretation: the results make a case for evaluating tools together and examining how configuration shapes coverage, rather than treating model choice alone as the answer. The findings come from a specific benchmark and do not establish how the tools perform across software outside it. How well would these differences in coverage and cost hold on vulnerabilities and applications outside the benchmark?","question":"How well would these differences in coverage and cost hold on vulnerabilities and applications outside the benchmark?","claims":[{"text":"The authors compare deterministic SAST and AI security agents using VulnBench 2.0, a benchmark of 108 attacker-reachable vulnerabilities in 20 open-source web applications.","excerpt":"VulnBench 2.0 is a benchmark of 108 hand-curated, attacker-reachable vulnerabilities in 20 open-source web applications","support":"source-stated"},{"text":"Snyk Code reached 82.4% macro recall and Claude Code with Opus 5 at xhigh effort reached 79.3%; the authors say their difference was too small to distinguish them on this benchmark.","excerpt":"Snyk Code reached 82.4% macro recall (89 of 108) and Claude Code with Opus 5 at xhigh effort 79.3% (85 of 108); they disagree on 28 vulnerabilities, split 16 to 12, too few to tell them apart.","support":"source-stated"},{"text":"Together, Snyk Code and Claude Code with Opus 5 at xhigh effort found 101 of the 108 vulnerabilities in the benchmark.","excerpt":"together they reached 101 of 108, the ceiling of all seven comparable configurations combined","support":"source-stated"},{"text":"AI configurations also reported at least seven real vulnerabilities that were absent from the reference set and were not reported by Snyk Code.","excerpt":"the AI configurations also reported at least seven real vulnerabilities that the reference set did not contain and Snyk Code did not report.","support":"source-stated"},{"text":"The authors report that results depend on the harness as well as the model, citing changes associated with review policy, line-level localization and file selection.","excerpt":"What an AI engine misses depends on its harness as much as on its model","support":"source-stated"},{"text":"The benchmark's findings and coverage comparisons do not establish how these tools perform on software outside the tested applications.","excerpt":"VulnBench 2.0 is a benchmark of 108 hand-curated, attacker-reachable vulnerabilities in 20 open-source web applications","support":"interpretation"}],"limitations":["The evaluation covers 108 hand-curated vulnerabilities in 20 open-source web applications, so its findings do not establish performance on software outside that benchmark.","The authors say the benchmark findings did not distinguish Snyk Code from Claude Code with Opus 5 at xhigh effort; reported performance should not be read as a general ranking.","Some cited policy results come from a supplementary run on 19 of 20 applications, as specified in the abstract."],"mediaId":"cc8eb88055d274e42ef06f39ab1d0fd3cf2f9d59e08eaaa2b6d2ea3d4947bcb3","correctionReason":null,"id":"f93051a53e268749548a10fdbc8c1ab437b1e84e3f93f9c380cd86bc2d1f822f","publicationId":"d2ebf4b4-c843-49c4-830d-bb48f7688952","version":1,"sourceUrl":"https://arxiv.org/abs/2610.38617","sourceTitle":"How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-28","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-30T06:42:57.356Z","createdAt":"2026-11-11T04:25:54.163Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"cc8eb88055d274e42ef06f39ab1d0fd3cf2f9d59e08eaaa2b6d2ea3d4947bcb3","kind":"article","url":"https://arxiv.org/abs/2610.38617","sourceUrl":"https://arxiv.org/abs/2610.38617","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-31T12:47:55.077Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.38617","checkedAt":"2026-11-11T04:56:08.979Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"cc8eb88055d274e42ef06f39ab1d0fd3cf2f9d59e08eaaa2b6d2ea3d4947bcb3","confidence":0.8899999999999999,"provider":"typesafe","model":"jev-1.13.0","inputHash":"215844944e0cf3c1daec8de802cf1505cc5054998e22380de268d4b0cf02b5a4","decidedAt":"2026-10-31T12:47:59.053Z"},"provenance":{"textHash":"cf7caf35af5c896209a3b3d6e80c3eab10ec1563bc748e5582404d7bbbfc7cf7","observationId":"2bbea6830393e1d65d4683296f47221de840b84e6300b969fe0a58e5688ea273","retrievedAt":"2026-10-31T12:47:50.596Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"ba103904-9fbd-4f8f-813f-edc0592aa012"},"reviewer":{"model":"gpt-6-luna","runId":"7d23ef82-0afe-4c2a-840c-cf89ba3c6ba8"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-11T04:25:54.163Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Tal, Liran","url":null,"kind":"unknown"},{"name":"Liu, Chris","url":null,"kind":"unknown"},{"name":"Fereday, Charlotte","url":null,"kind":"unknown"},{"name":"Roth, Sebastian","url":null,"kind":"unknown"},{"name":"Clark, Brian","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Tal, Liran","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Tal, Liran","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38617","locator":"head/meta[name=author|citation_author]"},"assertionId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T04:56:08.979Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8","sourceUrl":"https://arxiv.org/abs/2610.38617","contentSha256":"85089978de728c91a9bf83b0c026bb4ca8c26e785543f87adbb2b818750d7cca","textHash":"cf7caf35af5c896209a3b3d6e80c3eab10ec1563bc748e5582404d7bbbfc7cf7","observedAt":"2026-11-11T04:56:08.979Z"}},{"actor":{"name":"Liu, Chris","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liu, Chris","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38617","locator":"head/meta[name=author|citation_author]"},"assertionId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T04:56:08.979Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8","sourceUrl":"https://arxiv.org/abs/2610.38617","contentSha256":"85089978de728c91a9bf83b0c026bb4ca8c26e785543f87adbb2b818750d7cca","textHash":"cf7caf35af5c896209a3b3d6e80c3eab10ec1563bc748e5582404d7bbbfc7cf7","observedAt":"2026-11-11T04:56:08.979Z"}},{"actor":{"name":"Fereday, Charlotte","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Fereday, Charlotte","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38617","locator":"head/meta[name=author|citation_author]"},"assertionId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T04:56:08.979Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8","sourceUrl":"https://arxiv.org/abs/2610.38617","contentSha256":"85089978de728c91a9bf83b0c026bb4ca8c26e785543f87adbb2b818750d7cca","textHash":"cf7caf35af5c896209a3b3d6e80c3eab10ec1563bc748e5582404d7bbbfc7cf7","observedAt":"2026-11-11T04:56:08.979Z"}},{"actor":{"name":"Roth, Sebastian","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Roth, Sebastian","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38617","locator":"head/meta[name=author|citation_author]"},"assertionId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T04:56:08.979Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8","sourceUrl":"https://arxiv.org/abs/2610.38617","contentSha256":"85089978de728c91a9bf83b0c026bb4ca8c26e785543f87adbb2b818750d7cca","textHash":"cf7caf35af5c896209a3b3d6e80c3eab10ec1563bc748e5582404d7bbbfc7cf7","observedAt":"2026-11-11T04:56:08.979Z"}},{"actor":{"name":"Clark, Brian","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Clark, Brian","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38617","locator":"head/meta[name=author|citation_author]"},"assertionId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T04:56:08.979Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8","sourceUrl":"https://arxiv.org/abs/2610.38617","contentSha256":"85089978de728c91a9bf83b0c026bb4ca8c26e785543f87adbb2b818750d7cca","textHash":"cf7caf35af5c896209a3b3d6e80c3eab10ec1563bc748e5582404d7bbbfc7cf7","observedAt":"2026-11-11T04:56:08.979Z"}}],"participationCoverage":{"total":5,"returned":5,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.38617","snapshotId":"7a8aca140ebaae8da0de16aabc02f1eae35f83c1093dbcf28ba7136e6a938eb8","contentSha256":"85089978de728c91a9bf83b0c026bb4ca8c26e785543f87adbb2b818750d7cca","textHash":"cf7caf35af5c896209a3b3d6e80c3eab10ec1563bc748e5582404d7bbbfc7cf7","observedAt":"2026-11-11T04:56:08.979Z","basis":"publisher-page-metadata","sourceLinks":45,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/f93051a53e268749548a10fdbc8c1ab437b1e84e3f93f9c380cd86bc2d1f822f/source"}}}],"audits":[{"publicationId":"d2ebf4b4-c843-49c4-830d-bb48f7688952","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The draft names the five listed authors correctly. It keeps the benchmark results and the harness-versus-model argument attributed to them, and it labels the Escape Meridian interpretation separately.","excerpt":"Authors:Liran Tal, Chris Liu, Charlotte Fereday, Sebastian Roth, Brian Clark"},{"dimension":"summary","status":"uncertain","explanation":"The draft is mostly faithful, but the central term \"harness\" is never explained in plain language, even though the authors' main argument turns on it. Readers also get no explanation of \"xhigh effort.\" The examples help, but a newcomer may not see that a harness is the tool wrapper around the model, such as review policy and file selection. The question raises cost, which the summary never introduces. The title singles out Claude Code, while the study evaluated nine configurations. Smallest repair: add a short gloss or tooltip for harness, and either mention that coverage and cost were only loosely related or drop cost from the question.","excerpt":"What an AI engine misses depends on its harness as much as on its model"},{"dimension":"date","status":"pass","explanation":"The originalPublished date of 2026-10-28 matches the v1 submission date on arXiv. It is not the later feed date of 2026-10-31.","excerpt":"[Submitted on 28 Oct 2026]"},{"dimension":"support","status":"pass","explanation":"The source supports these figures: 82.4% and 79.3% recall, 28 disagreements too few to tell the tools apart, 101 of 108 found combined, and at least seven extra real vulnerabilities. The harness examples are also supported. The interpretation that tools should be evaluated together is labeled and grounded in the authors' statement that neither engine sufficed alone. The caveat about generalizing beyond the benchmark is appropriately conditional.","excerpt":"Neither engine was sufficient alone: together they reached 101 of 108, the ceiling of all seven comparable configurations combined"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct arXiv abstract page for this paper. It is the most direct access to the source.","excerpt":null}],"createdAt":"2026-11-11T04:25:54.163Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}