# SeVerus tests Rust verification agents by withdrawing proof context [Canonical reading](https://escapemeridian.com/readings/severus-tests-rust-verification-agents-by-withdrawing-proof-7a089bf0) The authors of SeVerus argue that existing benchmarks can make Verus verification look easier than it is in practice: they often provide auxiliary lemmas, or supporting pieces of a proof, that agents may need to discover and prove themselves. SeVerus organizes verification tasks as quests with levels, progressively removing proof context—from a level missing one lemma to one with no supporting lemmas. An independent checker rejects changes to executable code, weakened specifications, and unsound proof constructs. Applied to six open-source Verus projects, the method generated 819 quests and 10,095 levels. In experiments with GPT-6 Sol and GPT-6 Luna, and Claude Code using Sonnet 4.6, the authors report that success decreased and cost increased as proof context was withdrawn. The strongest agent solved the final level of 52 of 60 sampled quests, but could reliably solve only 35% of their levels; failures were concentrated in longer quests. The authors also report that successful GPT-6 Sol proofs rarely reproduced withheld lemmas, instead using alternative proof decompositions. The results show how performance changes as these benchmarks remove help; the abstract does not explain why failures were concentrated in longer quests. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Nov 4, 2026 (2026-11-04). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-08T18:09:31.015Z. First observed: 2026-11-06T05:07:32.056Z. Retrieved: 2026-11-08T18:08:43.373Z. - [SeVerus: Context-Aware, Gamified Benchmarking for Rust Verification](https://arxiv.org/abs/2611.07194) ## Limits The reported results are from experiments on benchmarks generated from six open-source Verus projects; the abstract does not report how performance on these levels relates to performance on other verification work. The abstract reports that failures were concentrated in longer quests but does not identify what features of those quests account for that pattern. The reported 35% is the strongest agent's reliably solved share of sampled quest levels, not a measure of real-world verification reliability. ## Open question What features of the longer quests account for the concentration of failures the authors report? ## Source-stated The authors say existing Verus benchmarks typically supply auxiliary lemmas, while practical verification may require agents to discover and prove them. > These benchmarks typically provide the auxiliary lemmas needed to complete a proof. In practice, however, such lemmas must be discovered and proved. - [Supporting source](https://arxiv.org/abs/2611.07194) ## Source-stated SeVerus creates quest levels by progressively removing proof context, from requiring one missing lemma to proving the target without supporting lemmas. > These levels range from the easiest, requiring completing only a single missing lemma, to the hardest, requiring proving the target without any supporting lemmas. - [Supporting source](https://arxiv.org/abs/2611.07194) ## Source-stated An independent checker rejects changes to executable code, weakened specifications, and unsound proof constructs. > An independent checker rejects changes to executable code, weakened specifications, and unsound proof constructs. - [Supporting source](https://arxiv.org/abs/2611.07194) ## Source-stated Applied to six open-source Verus projects, SeVerus generated 819 quests comprising 10,095 levels. > Applied to six open-source Verus projects, SEVERUS generates 819 quests comprising 10,095 levels. - [Supporting source](https://arxiv.org/abs/2611.07194) ## Source-stated The authors report that agent success decreased and cost increased as proof context was withdrawn; the strongest agent solved the final level of 52 of 60 sampled quests but reliably solved 35% of their levels. > The strongest agent solves the final level of 52 of 60 sampled quests, yet can only reliably solve 35% of their levels, with failures concentrated in longer quests. - [Supporting source](https://arxiv.org/abs/2611.07194) ## Source-stated The authors report that successful GPT-6 Sol proofs rarely reproduced withheld lemmas and instead used alternative proof decompositions. > Successful GPT-6 Sol proofs rarely reproduce the withheld lemmas, instead, using alternative proof decompositions. - [Supporting source](https://arxiv.org/abs/2611.07194) ## Source and contributor context Silva, Álvaro F., Afonso, Martim, Saavedra, Nuno, First, Emily, Mendes, Alexandra, Ferreira, João F. · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## GPT-6 Sol GPT-6 Sol is an OpenAI model announced on September 22, 2026 as a lower-cost tier trained with methods similar to Astra. - [Explore this context](/entities/gpt-6-sol) - [Reference](https://openai.com/index/introducing-gpt-6-sol-and-luna) ## GPT-6 Luna GPT-6 Luna is an OpenAI model announced on September 22, 2026 as a cheaper GPT-6 tier. - [Explore this context](/entities/gpt-6-luna) - [Reference](https://openai.com/index/introducing-gpt-6-sol-and-luna) ## Claude Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding. - [Explore this context](/entities/claude) - [Reference](https://claude.com/product/overview) - [Reference](https://www.anthropic.com/news) - [Reference](https://www.anthropic.com/research/team/economics) ## Editorial provenance Version 1. Publication ID: 90389ee8-c662-49de-8320-2098a3c4f76d. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-08T18:09:31.015Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 8408223965b681653dd00a99415ba08895945a0de02717d02627d2c13ec09de4. Source text hash: 4fb792e66efb5df94685b72f40017a5f7a9638930401588b527899b1fd6d6f35. ## Revision and source-check history ### Version 1 — 2026-11-08T18:09:31.015Z The authors of SeVerus argue that existing benchmarks can make Verus verification look easier than it is in practice: they often provide auxiliary lemmas, or supporting pieces of a proof, that agents may need to discover and prove themselves. SeVerus organizes verification tasks as quests with levels, progressively removing proof context—from a level missing one lemma to one with no supporting lemmas. An independent checker rejects changes to executable code, weakened specifications, and unsound proof constructs. Applied to six open-source Verus projects, the method generated 819 quests and 10,095 levels. In experiments with GPT-6 Sol and GPT-6 Luna, and Claude Code using Sonnet 4.6, the authors report that success decreased and cost increased as proof context was withdrawn. The strongest agent solved the final level of 52 of 60 sampled quests, but could reliably solve only 35% of their levels; failures were concentrated in longer quests. The authors also report that successful GPT-6 Sol proofs rarely reproduced withheld lemmas, instead using alternative proof decompositions. The results show how performance changes as these benchmarks remove help; the abstract does not explain why failures were concentrated in longer quests. ## Original source arXiv cs.AI · Article Original source date: Nov 4, 2026 (2026-11-04). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-08T18:09:31.015Z. First observed: 2026-11-06T05:07:32.056Z. Retrieved: 2026-11-08T18:08:43.373Z. - [SeVerus: Context-Aware, Gamified Benchmarking for Rust Verification](https://arxiv.org/abs/2611.07194) ## Limits The reported results are from experiments on benchmarks generated from six open-source Verus projects; the abstract does not report how performance on these levels relates to performance on other verification work. The abstract reports that failures were concentrated in longer quests but does not identify what features of those quests account for that pattern. The reported 35% is the strongest agent's reliably solved share of sampled quest levels, not a measure of real-world verification reliability. ## Open question What features of the longer quests account for the concentration of failures the authors report? ## Source-stated The authors say existing Verus benchmarks typically supply auxiliary lemmas, while practical verification may require agents to discover and prove them. > These benchmarks typically provide the auxiliary lemmas needed to complete a proof. In practice, however, such lemmas must be discovered and proved. - [Supporting source](https://arxiv.org/abs/2611.07194) ## Source-stated SeVerus creates quest levels by progressively removing proof context, from requiring one missing lemma to proving the target without supporting lemmas. > These levels range from the easiest, requiring completing only a single missing lemma, to the hardest, requiring proving the target without any supporting lemmas. - [Supporting source](https://arxiv.org/abs/2611.07194) ## Source-stated An independent checker rejects changes to executable code, weakened specifications, and unsound proof constructs. > An independent checker rejects changes to executable code, weakened specifications, and unsound proof constructs. - [Supporting source](https://arxiv.org/abs/2611.07194) ## Source-stated Applied to six open-source Verus projects, SeVerus generated 819 quests comprising 10,095 levels. > Applied to six open-source Verus projects, SEVERUS generates 819 quests comprising 10,095 levels. - [Supporting source](https://arxiv.org/abs/2611.07194) ## Source-stated The authors report that agent success decreased and cost increased as proof context was withdrawn; the strongest agent solved the final level of 52 of 60 sampled quests but reliably solved 35% of their levels. > The strongest agent solves the final level of 52 of 60 sampled quests, yet can only reliably solve 35% of their levels, with failures concentrated in longer quests. - [Supporting source](https://arxiv.org/abs/2611.07194) ## Source-stated The authors report that successful GPT-6 Sol proofs rarely reproduced withheld lemmas and instead used alternative proof decompositions. > Successful GPT-6 Sol proofs rarely reproduce the withheld lemmas, instead, using alternative proof decompositions. - [Supporting source](https://arxiv.org/abs/2611.07194) ## Source and contributor context Silva, Álvaro F., Afonso, Martim, Saavedra, Nuno, First, Emily, Mendes, Alexandra, Ferreira, João F. · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## GPT-6 Sol GPT-6 Sol is an OpenAI model announced on September 22, 2026 as a lower-cost tier trained with methods similar to Astra. - [Explore this context](/entities/gpt-6-sol) - [Reference](https://openai.com/index/introducing-gpt-6-sol-and-luna) ## GPT-6 Luna GPT-6 Luna is an OpenAI model announced on September 22, 2026 as a cheaper GPT-6 tier. - [Explore this context](/entities/gpt-6-luna) - [Reference](https://openai.com/index/introducing-gpt-6-sol-and-luna) ## Claude Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding. - [Explore this context](/entities/claude) - [Reference](https://claude.com/product/overview) - [Reference](https://www.anthropic.com/news) - [Reference](https://www.anthropic.com/research/team/economics) ## Editorial provenance Version 1. Publication ID: 90389ee8-c662-49de-8320-2098a3c4f76d. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-08T18:09:31.015Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 8408223965b681653dd00a99415ba08895945a0de02717d02627d2c13ec09de4. Source text hash: 4fb792e66efb5df94685b72f40017a5f7a9638930401588b527899b1fd6d6f35. ### uncertain — 2026-11-08T18:09:31.015Z — anthropic/claude-opus-5.5 Publication: 90389ee8-c662-49de-8320-2098a3c4f76d attribution — pass: The claims are attributed to the SeVerus authors and framed as their reports. The draft drops 'Codex' as the harness for GPT-6 Sol and GPT-6 Luna, writing only 'experiments with GPT-6 Sol and GPT-6 Luna'. That is a small loss of precision, not a misattribution. Smallest repair: write 'Codex using GPT-6 Sol and GPT-6 Luna'. > Experiments with Codex using GPT-6 Sol and GPT-6 Luna, and with Claude Code using Sonnet 4.6 summary — uncertain: The draft explains lemmas but never says what Verus is or what verification means. A non-specialist cannot tell that Verus checks mathematical proofs that Rust programs meet their specifications. Terms like proof context, unsound proof constructs and 'reliably solve' are also left undefined; the abstract ties 'reliably' to a noisy binary search. The draft also drops the authors' stated motivation: agents score highly on existing benchmarks, and SeVerus aims to keep challenging more capable agents. Smallest repair: add a plain-language gloss of Verus and verification, and restore that motivation. > Coding agents achieve high success rates on existing benchmarks for Verus, a verifier for Rust programs. date — pass: The originalPublished date of 2026-11-04 matches the v1 submission date on arXiv. It is not the later feed date. > \[Submitted on 4 Nov 2026\] support — pass: Each claim matches the abstract verbatim in substance, including the 819 quests, 10,095 levels, 52 of 60 sampled quests, 35% of levels and the alternative proof decompositions. The limitations properly separate the 35% figure from real-world reliability. The question about longer quests follows from an issue the abstract leaves unexplained, and no benefits or costs are invented. > The strongest agent solves the final level of 52 of 60 sampled quests, yet can only reliably solve 35% of their levels, with failures concentrated in longer quests. media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the original source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/7a089bf0e5f4a607480ce2d4992d9c30d2df82b55329e5f1f5ee6d7a4bb099ef/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/severus-tests-rust-verification-agents-by-withdrawing-proof-7a089bf0.json)