# arXiv paper reports ARMs attack success above 90% on Claude-4-Sonnet [Canonical reading](https://escapemeridian.com/readings/arxiv-paper-reports-arms-attack-success-above-90-on-claude-4-6d1dbf3f) In their paper, Zhaorun Chen and coauthors present ARMs, an agent for testing vision-language models (VLMs) against harmful outputs. Given a target behavior or risk, ARMs adapts red-teaming strategies and combines 17 algorithms with 11 proposed multimodal attack strategies. The authors say layered memory and an epsilon-greedy exploration algorithm help balance attack diversity and effectiveness. On instance- and policy-based benchmarks, they report attack success rates averaging 52.1% above baselines and exceeding 90% on Claude-4-Sonnet. They also say ARMs generated more diverse test instances, which they describe as revealing emerging vulnerabilities. Using those instances, the authors built ARMs-Bench, a dataset of over 30K examples across 51 risk categories. They report that safety fine-tuning with it improves VLM robustness while preserving general utility. The paper’s approach is to automate varied tests rather than rely on a narrow set of patterns or extensive manual engineering. The reported gains concern the paper’s benchmarks and fine-tuning experiments; the supplied abstract does not detail how those results translate to deployed systems. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Oct 3, 2025 (2025-10-03). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-08T04:32:30.460Z. First observed: 2026-11-06T05:03:35.401Z. Retrieved: 2026-11-08T04:31:40.113Z. - [ARMs: Adaptive Red-Teaming Agent against Multimodal Models with Plug-and-Play Attacks](https://arxiv.org/abs/2510.02677) ## Limits The supplied abstract reports benchmark and fine-tuning results but does not provide the underlying evaluation details or explain how results translate to deployed systems. The authors describe ARMs-Bench fine-tuning as improving robustness while preserving general utility; the abstract does not specify the measures used for those outcomes. ## Open question What measures and results support the authors’ claim that safety fine-tuning with ARMs-Bench improves robustness while preserving general utility? ## Source-stated The authors propose ARMs, an agent that adapts red-teaming strategies to a specified harmful behavior or risk in vision-language models. > Given a target harmful behavior or risk definition, ARMs automatically optimizes diverse red-teaming strategies - [Supporting source](https://arxiv.org/abs/2510.02677) ## Source-stated The authors say ARMs combines 11 proposed multimodal attack strategies with 17 red-teaming algorithms. > We propose 11 novel multimodal attack strategies - [Supporting source](https://arxiv.org/abs/2510.02677) ## Source-stated On instance- and policy-based benchmarks, the authors report attack success rates averaging 52.1% above baselines and exceeding 90% on Claude-4-Sonnet. > achieves SOTA attack success rates, exceeding baselines by an average of 52.1% and surpassing 90% on Claude-4-Sonnet - [Supporting source](https://arxiv.org/abs/2510.02677) ## Source-stated The authors report that ARMs-Bench contains over 30K red-teaming instances across 51 risk categories, and that safety fine-tuning with it improves robustness while preserving general utility. > comprising over 30K red-teaming instances spanning 51 diverse risk categories - [Supporting source](https://arxiv.org/abs/2510.02677) ## Source-stated The proposed approach aims to balance attack diversity and effectiveness through layered memory and epsilon-greedy exploration. > To balance the diversity and effectiveness of the attack, we design a layered memory with an epsilon-greedy attack exploration algorithm. - [Supporting source](https://arxiv.org/abs/2510.02677) ## Source and contributor context Chen, Zhaorun, Liu, Xun, Kang, Mintong, Zhang, Jiawei, Pan, Minzhou, Yang, Shuang, Li, Bo · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Red-teaming Deliberately attacking or tricking a system to find its weaknesses before others do. - [Definition reference](https://en.wikipedia.org/wiki/Red_team) ## Multimodal model A model that works with more than one kind of input or output, such as text, images and audio. - [Definition reference](https://developers.google.com/machine-learning/glossary#multimodal-model) ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Fine-tuning Further training of an existing model to adapt it to a particular task or use. - [Definition reference](https://developers.google.com/machine-learning/glossary#fine-tuning) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Claude Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding. - [Explore this context](/entities/claude) - [Reference](https://claude.com/product/overview) - [Reference](https://www.anthropic.com/news) - [Reference](https://www.anthropic.com/research/team/economics) ## Editorial provenance Version 1. Publication ID: e3cae356-876c-4194-808c-9e1d84ca87cf. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-08T04:32:30.460Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 4893ad348df38fba8e86e36b31b85cfed5e96b121cefa7615b1a16302d9dd771. Source text hash: 1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc. ## Revision and source-check history ### Version 1 — 2026-11-08T04:32:30.460Z In their paper, Zhaorun Chen and coauthors present ARMs, an agent for testing vision-language models (VLMs) against harmful outputs. Given a target behavior or risk, ARMs adapts red-teaming strategies and combines 17 algorithms with 11 proposed multimodal attack strategies. The authors say layered memory and an epsilon-greedy exploration algorithm help balance attack diversity and effectiveness. On instance- and policy-based benchmarks, they report attack success rates averaging 52.1% above baselines and exceeding 90% on Claude-4-Sonnet. They also say ARMs generated more diverse test instances, which they describe as revealing emerging vulnerabilities. Using those instances, the authors built ARMs-Bench, a dataset of over 30K examples across 51 risk categories. They report that safety fine-tuning with it improves VLM robustness while preserving general utility. The paper’s approach is to automate varied tests rather than rely on a narrow set of patterns or extensive manual engineering. The reported gains concern the paper’s benchmarks and fine-tuning experiments; the supplied abstract does not detail how those results translate to deployed systems. ## Original source arXiv cs.AI · Article Original source date: Oct 3, 2025 (2025-10-03). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-08T04:32:30.460Z. First observed: 2026-11-06T05:03:35.401Z. Retrieved: 2026-11-08T04:31:40.113Z. - [ARMs: Adaptive Red-Teaming Agent against Multimodal Models with Plug-and-Play Attacks](https://arxiv.org/abs/2510.02677) ## Limits The supplied abstract reports benchmark and fine-tuning results but does not provide the underlying evaluation details or explain how results translate to deployed systems. The authors describe ARMs-Bench fine-tuning as improving robustness while preserving general utility; the abstract does not specify the measures used for those outcomes. ## Open question What measures and results support the authors’ claim that safety fine-tuning with ARMs-Bench improves robustness while preserving general utility? ## Source-stated The authors propose ARMs, an agent that adapts red-teaming strategies to a specified harmful behavior or risk in vision-language models. > Given a target harmful behavior or risk definition, ARMs automatically optimizes diverse red-teaming strategies - [Supporting source](https://arxiv.org/abs/2510.02677) ## Source-stated The authors say ARMs combines 11 proposed multimodal attack strategies with 17 red-teaming algorithms. > We propose 11 novel multimodal attack strategies - [Supporting source](https://arxiv.org/abs/2510.02677) ## Source-stated On instance- and policy-based benchmarks, the authors report attack success rates averaging 52.1% above baselines and exceeding 90% on Claude-4-Sonnet. > achieves SOTA attack success rates, exceeding baselines by an average of 52.1% and surpassing 90% on Claude-4-Sonnet - [Supporting source](https://arxiv.org/abs/2510.02677) ## Source-stated The authors report that ARMs-Bench contains over 30K red-teaming instances across 51 risk categories, and that safety fine-tuning with it improves robustness while preserving general utility. > comprising over 30K red-teaming instances spanning 51 diverse risk categories - [Supporting source](https://arxiv.org/abs/2510.02677) ## Source-stated The proposed approach aims to balance attack diversity and effectiveness through layered memory and epsilon-greedy exploration. > To balance the diversity and effectiveness of the attack, we design a layered memory with an epsilon-greedy attack exploration algorithm. - [Supporting source](https://arxiv.org/abs/2510.02677) ## Source and contributor context Chen, Zhaorun, Liu, Xun, Kang, Mintong, Zhang, Jiawei, Pan, Minzhou, Yang, Shuang, Li, Bo · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Red-teaming Deliberately attacking or tricking a system to find its weaknesses before others do. - [Definition reference](https://en.wikipedia.org/wiki/Red_team) ## Multimodal model A model that works with more than one kind of input or output, such as text, images and audio. - [Definition reference](https://developers.google.com/machine-learning/glossary#multimodal-model) ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Fine-tuning Further training of an existing model to adapt it to a particular task or use. - [Definition reference](https://developers.google.com/machine-learning/glossary#fine-tuning) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Claude Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding. - [Explore this context](/entities/claude) - [Reference](https://claude.com/product/overview) - [Reference](https://www.anthropic.com/news) - [Reference](https://www.anthropic.com/research/team/economics) ## Editorial provenance Version 1. Publication ID: e3cae356-876c-4194-808c-9e1d84ca87cf. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-08T04:32:30.460Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 4893ad348df38fba8e86e36b31b85cfed5e96b121cefa7615b1a16302d9dd771. Source text hash: 1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc. ### passed — 2026-11-08T04:32:30.460Z — anthropic/claude-opus-5.5 Publication: e3cae356-876c-4194-808c-9e1d84ca87cf attribution — pass: The reading attributes the paper to Zhaorun Chen and coauthors and keeps the performance and fine-tuning results framed as the authors' reports. This matches the arXiv author list and abstract. > Authors:Zhaorun Chen, Xun Liu, Mintong Kang, Jiawei Zhang, Minzhou Pan, Shuang Yang, Bo Li summary — pass: The reading names the actor, spells out vision-language models (VLMs), and describes ARMs as an automated way to test whether VLMs can be pushed into harmful outputs. It covers the method, results, dataset and fine-tuning claim, and notes that the evidence comes from benchmarks. Epsilon-greedy is not explained, but it is secondary and could get a term tooltip. The title and question follow from the abstract. > we propose ARMs, an adaptive red-teaming agent that systematically conducts comprehensive risk assessments for VLMs date — pass: The originalPublished date of 2025-10-03 matches the v1 submission date. It is not the later v2 revision date of 4 Nov 2026 or the 2026 feed date. > \[Submitted on 3 Oct 2025 (v1), last revised 4 Nov 2026 (this version, v2)\] support — pass: Every claim is stated in the abstract: the 11 strategies, the 17 integrated algorithms, layered memory with epsilon-greedy exploration, the 52.1% average gain over baselines and over 90% on Claude-4-Sonnet, greater instance diversity, ARMs-Bench at over 30K instances across 51 categories, and the fine-tuning outcome. The limitations accurately note that the abstract lacks evaluation detail and evidence about deployed systems, without overreaching. > Safety fine-tuning with ARMs-Bench substantially improves the robustness of VLMs while preserving their general utility media — pass: The selected media is the verified direct link to the arXiv abstract page for this paper, which is the most direct access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/6d1dbf3faf9f0cf48f803e2a92ebaaf1e5368532e7148a2c8d8484d01920f844/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-paper-reports-arms-attack-success-above-90-on-claude-4-6d1dbf3f.json)