# Hugging Face: CWE-bench v2: More coverage, quality, and transparency [Canonical reading](https://escapemeridian.com/readings/hugging-face-cwe-bench-v2-more-coverage-quality-and-transparency-7112f2d4) Collinear AI authors Gonzalo Gonzalez, Ajay Jagannath, Nazneen Rajani, and Nathan Habib present CWE-bench v2, a public benchmark for testing whether models can find and fix security weaknesses in real open-source repositories. They say its 100 tasks cover 91 different CWEs, with deterministic scoring and hand-reviewed tasks; they also describe checks intended to make instructions and verifiers realistic and fair. The authors argue that broad coverage matters because defenders must anticipate many ways an attacker could get in, not just recognize familiar patterns. In their reported results, Gemini 4 Argon leads the evaluated models, but the authors emphasize that even this leader fails 19% of the set on every attempt. They describe that remaining long tail as far from solved. These benchmark scores show performance on this task set, not how reliably models find or fix vulnerabilities in deployed software. The public set, grader, and harness are open, and the authors say the same tasks can provide rewards for training defensive agents. The held-out set, which models have not seen, is available for evaluation by contacting Collinear AI. How well do models that perform on CWE-bench v2 transfer to security weaknesses in repositories or conditions outside the benchmark? Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source Hugging Face · Article Original source date: Nov 3, 2026 (2026-11-03T00:00:00.764Z). Date basis: Date stated by the publisher. Added to Escape Meridian: 2026-11-10T09:53:41.961Z. First observed: 2026-11-03T17:25:19.852Z. Retrieved: 2026-11-03T17:32:28.988Z. - [CWE-bench v2: More coverage, quality, and transparency](https://huggingface.co/blog/cwe-bench) ## Limits The reported scores measure performance on this benchmark; the article does not report how well they predict security performance in deployed software. Evaluation on the held-out set requires contacting cyber@collinear.ai, so the article does not provide held-out results. The article presents the authors’ description of task quality checks and results; it does not provide independent validation of those claims. ## Open question How well do models that perform on CWE-bench v2 transfer to security weaknesses in repositories or conditions outside the benchmark? ## Source-stated The authors say CWE-bench v2 makes 100 tasks public, covering 91 different CWEs. > CWE-bench v2 opens up a public set of 100 tasks that anyone can run and evaluate their models on. - [Supporting source](https://huggingface.co/blog/cwe-bench) ## Source-stated The authors say every task was reviewed by hand and that they checked whether instructions and verifiers were realistic and fair. > Every task was also reviewed by hand. Open-sourcing the set is part of the same effort to improve quality. - [Supporting source](https://huggingface.co/blog/cwe-bench) ## Source-stated The authors report that Gemini 4 Argon leads the 11 evaluated models, while still failing 19% of the set on every attempt. > Of the 11 models evaluated, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4 - [Supporting source](https://huggingface.co/blog/cwe-bench) ## Source-stated The authors say the task environments and verifiers can provide rewards for training defensive agents as well as evaluating them. > the same environments and verifiers can provide rewards for training defensive agents, not just for evaluating them. - [Supporting source](https://huggingface.co/blog/cwe-bench) ## Source and contributor context Gonzalo Gonzalez, Ajay Jagannath, Nazneen Rajani, Nathan Habib · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Held-out set Data kept aside during training and used only to test how well a model works on examples it has not seen. - [Definition reference](https://developers.google.com/machine-learning/glossary#holdout-data) ## Hugging Face Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems. - [Explore this context](/entities/hugging-face) - [Reference](https://huggingface.co/blog) ## Editorial provenance Version 1. Publication ID: b8755caf-1970-40fb-8639-01f6949f4eaa. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T09:53:41.961Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: aa560c3d3200f269127d6a1b2ab6e4584773a0502785e08d346153fc30e5120b. Source text hash: 07d574caa2f94a1023cf1b481cfb1db5fd59012564a4d4c9f026dd8f89101efd. ## Revision and source-check history ### Version 1 — 2026-11-10T09:53:41.961Z Collinear AI authors Gonzalo Gonzalez, Ajay Jagannath, Nazneen Rajani, and Nathan Habib present CWE-bench v2, a public benchmark for testing whether models can find and fix security weaknesses in real open-source repositories. They say its 100 tasks cover 91 different CWEs, with deterministic scoring and hand-reviewed tasks; they also describe checks intended to make instructions and verifiers realistic and fair. The authors argue that broad coverage matters because defenders must anticipate many ways an attacker could get in, not just recognize familiar patterns. In their reported results, Gemini 4 Argon leads the evaluated models, but the authors emphasize that even this leader fails 19% of the set on every attempt. They describe that remaining long tail as far from solved. These benchmark scores show performance on this task set, not how reliably models find or fix vulnerabilities in deployed software. The public set, grader, and harness are open, and the authors say the same tasks can provide rewards for training defensive agents. The held-out set, which models have not seen, is available for evaluation by contacting Collinear AI. How well do models that perform on CWE-bench v2 transfer to security weaknesses in repositories or conditions outside the benchmark? ## Original source Hugging Face · Article Original source date: Nov 3, 2026 (2026-11-03T00:00:00.764Z). Date basis: Date stated by the publisher. Added to Escape Meridian: 2026-11-10T09:53:41.961Z. First observed: 2026-11-03T17:25:19.852Z. Retrieved: 2026-11-03T17:32:28.988Z. - [CWE-bench v2: More coverage, quality, and transparency](https://huggingface.co/blog/cwe-bench) ## Limits The reported scores measure performance on this benchmark; the article does not report how well they predict security performance in deployed software. Evaluation on the held-out set requires contacting cyber@collinear.ai, so the article does not provide held-out results. The article presents the authors’ description of task quality checks and results; it does not provide independent validation of those claims. ## Open question How well do models that perform on CWE-bench v2 transfer to security weaknesses in repositories or conditions outside the benchmark? ## Source-stated The authors say CWE-bench v2 makes 100 tasks public, covering 91 different CWEs. > CWE-bench v2 opens up a public set of 100 tasks that anyone can run and evaluate their models on. - [Supporting source](https://huggingface.co/blog/cwe-bench) ## Source-stated The authors say every task was reviewed by hand and that they checked whether instructions and verifiers were realistic and fair. > Every task was also reviewed by hand. Open-sourcing the set is part of the same effort to improve quality. - [Supporting source](https://huggingface.co/blog/cwe-bench) ## Source-stated The authors report that Gemini 4 Argon leads the 11 evaluated models, while still failing 19% of the set on every attempt. > Of the 11 models evaluated, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4 - [Supporting source](https://huggingface.co/blog/cwe-bench) ## Source-stated The authors say the task environments and verifiers can provide rewards for training defensive agents as well as evaluating them. > the same environments and verifiers can provide rewards for training defensive agents, not just for evaluating them. - [Supporting source](https://huggingface.co/blog/cwe-bench) ## Source and contributor context Gonzalo Gonzalez, Ajay Jagannath, Nazneen Rajani, Nathan Habib · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Held-out set Data kept aside during training and used only to test how well a model works on examples it has not seen. - [Definition reference](https://developers.google.com/machine-learning/glossary#holdout-data) ## Hugging Face Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems. - [Explore this context](/entities/hugging-face) - [Reference](https://huggingface.co/blog) ## Editorial provenance Version 1. Publication ID: b8755caf-1970-40fb-8639-01f6949f4eaa. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T09:53:41.961Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: aa560c3d3200f269127d6a1b2ab6e4584773a0502785e08d346153fc30e5120b. Source text hash: 07d574caa2f94a1023cf1b481cfb1db5fd59012564a4d4c9f026dd8f89101efd. ### uncertain — 2026-11-10T09:53:41.961Z — anthropic/claude-opus-5.5 Publication: b8755caf-1970-40fb-8639-01f6949f4eaa attribution — uncertain: The draft calls all four authors 'Collinear AI authors.' The byline shows a collinear-ai affiliation after Gonzalez, Jagannath and Rajani, but not after Nathan Habib (SaylorTwift). His affiliation is not stated, so labeling him a Collinear AI author goes beyond the evidence. Smallest repair: 'Gonzalo Gonzalez, Ajay Jagannath and Nazneen Rajani of Collinear AI, with Nathan Habib.' The 'Hugging Face:' title prefix correctly names the publisher. > Nazneen Rajani nazneen collinear-ai Nathan Habib SaylorTwift summary — uncertain: The reading is otherwise faithful and keeps the authors' force: exhaustiveness, the defender framing and the 'far from solved' long tail. However, the central term 'CWEs' (Common Weakness Enumeration categories of software security weaknesses) is never explained, and a general reader cannot tell what '91 different CWEs' means. A term-library tooltip for CWE is needed; 'verifier' would also benefit from one. The question is substantive and not leading. > These 100 tasks cover 91 different CWEs, spanning a wide range of vulnerability classes. date — pass: The originalPublished value, 2026-11-03, matches the page's stated publication date of November 3, 2026. No earlier first-publication date is indicated. > Published November 3, 2026 support — pass: The claims are source-stated: 100 public tasks covering 91 CWEs, hand review, quality checks, Gemini 4 Argon leading 11 models while failing 19% on every attempt, and RL reward use. The caveat that the scores do not show deployed-software reliability is appropriately framed. The limitations accurately note that no held-out results or independent validation are given, and no benefits or costs are invented. > However, even the leader fails 19% of the set on every attempt, so the long tail is far from solved. media — pass: The selected media is the verified direct link to the original Hugging Face blog post, which is the most relevant way to access the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/7112f2d4c83cb1b03939ecf78d93be1fa2f364ae68f21e8dead44f5447c56ac0/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/hugging-face-cwe-bench-v2-more-coverage-quality-and-transparency-7112f2d4.json)