Chen, Zhaorun, Liu, Xun, Kang, Mintong, Zhang, Jiawei, Pan, Minzhou, Yang, Shuang, Li, Bo · Oct 3, 2025
arXiv paper reports ARMs attack success above 90% on Claude-4-Sonnet
Markdown · JSON · Original source
In their paper, Zhaorun Chen and coauthors present ARMs, an agent for testing vision-language models (VLMs) against harmful outputs. Given a target behavior or risk, ARMs adapts strategies and combines 17 algorithms with 11 proposed attack strategies. The authors say layered memory and an epsilon-greedy exploration algorithm help balance attack diversity and effectiveness. On instance- and policy-based , they report attack success rates averaging 52.1% above baselines and exceeding 90% on -4-Sonnet. They also say ARMs generated more diverse test instances, which they describe as revealing emerging vulnerabilities. Using those instances, the authors built ARMs-Bench, a dataset of over 30K examples across 51 risk categories. They report that safety with it improves VLM robustness while preserving general utility. The paper’s approach is to automate varied tests rather than rely on a narrow set of patterns or extensive manual engineering. The reported gains concern the paper’s benchmarks and fine-tuning experiments; the supplied abstract does not detail how those results translate to deployed systems.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 3, 2025 (2025-10-03).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-08T04:32:30.460Z. First observed: 2026-11-06T05:03:35.401Z. Retrieved: 2026-11-08T04:31:40.113Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The supplied abstract reports and results but does not provide the underlying evaluation details or explain how results translate to deployed systems. The authors describe ARMs-Bench fine-tuning as improving robustness while preserving general utility; the abstract does not specify the measures used for those outcomes.
Open question
What measures and results support the authors’ claim that safety with ARMs-Bench improves robustness while preserving general utility?
Source-stated
The authors propose ARMs, an agent that adapts red-teaming strategies to a specified harmful behavior or risk in vision-language models.
Given a target harmful behavior or risk definition, ARMs automatically optimizes diverse red-teaming strategies
Source-stated
The authors say ARMs combines 11 proposed multimodal attack strategies with 17 red-teaming algorithms.
We propose 11 novel multimodal attack strategies
Source-stated
On instance- and policy-based benchmarks, the authors report attack success rates averaging 52.1% above baselines and exceeding 90% on Claude-4-Sonnet.
achieves SOTA attack success rates, exceeding baselines by an average of 52.1% and surpassing 90% on Claude-4-Sonnet
Source-stated
The authors report that ARMs-Bench contains over 30K red-teaming instances across 51 risk categories, and that safety fine-tuning with it improves robustness while preserving general utility.
comprising over 30K red-teaming instances spanning 51 diverse risk categories
Source-stated
The proposed approach aims to balance attack diversity and effectiveness through layered memory and epsilon-greedy exploration.
To balance the diversity and effectiveness of the attack, we design a layered memory with an epsilon-greedy attack exploration algorithm.
Source and contributor context
Chen, Zhaorun, Liu, Xun, Kang, Mintong, Zhang, Jiawei, Pan, Minzhou, Yang, Shuang, Li, Bo · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Red-teaming
Deliberately attacking or tricking a system to find its weaknesses before others do.
Multimodal model
A model that works with more than one kind of input or output, such as text, images and audio.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Fine-tuning
Further training of an existing model to adapt it to a particular task or use.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: e3cae356-876c-4194-808c-9e1d84ca87cf.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-08T04:32:30.460Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 4893ad348df38fba8e86e36b31b85cfed5e96b121cefa7615b1a16302d9dd771. Source text hash: 1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc.
Revision and source-check history
Version 1 · 2026-11-08T04:32:30.460Z
arXiv paper reports ARMs attack success above 90% on Claude-4-Sonnet
In their paper, Zhaorun Chen and coauthors present ARMs, an agent for testing vision-language models (VLMs) against harmful outputs. Given a target behavior or risk, ARMs adapts strategies and combines 17 algorithms with 11 proposed attack strategies. The authors say layered memory and an epsilon-greedy exploration algorithm help balance attack diversity and effectiveness. On instance- and policy-based , they report attack success rates averaging 52.1% above baselines and exceeding 90% on -4-Sonnet. They also say ARMs generated more diverse test instances, which they describe as revealing emerging vulnerabilities. Using those instances, the authors built ARMs-Bench, a dataset of over 30K examples across 51 risk categories. They report that safety with it improves VLM robustness while preserving general utility. The paper’s approach is to automate varied tests rather than rely on a narrow set of patterns or extensive manual engineering. The reported gains concern the paper’s benchmarks and fine-tuning experiments; the supplied abstract does not detail how those results translate to deployed systems.
Original source
cs.AI · Article
Source details
Original source date: Oct 3, 2025 (2025-10-03).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-08T04:32:30.460Z. First observed: 2026-11-06T05:03:35.401Z. Retrieved: 2026-11-08T04:31:40.113Z.
Limits
The supplied abstract reports and results but does not provide the underlying evaluation details or explain how results translate to deployed systems. The authors describe ARMs-Bench fine-tuning as improving robustness while preserving general utility; the abstract does not specify the measures used for those outcomes.
Open question
What measures and results support the authors’ claim that safety with ARMs-Bench improves robustness while preserving general utility?
Source-stated
The authors propose ARMs, an agent that adapts red-teaming strategies to a specified harmful behavior or risk in vision-language models.
Given a target harmful behavior or risk definition, ARMs automatically optimizes diverse red-teaming strategies
Source-stated
The authors say ARMs combines 11 proposed multimodal attack strategies with 17 red-teaming algorithms.
We propose 11 novel multimodal attack strategies
Source-stated
On instance- and policy-based benchmarks, the authors report attack success rates averaging 52.1% above baselines and exceeding 90% on Claude-4-Sonnet.
achieves SOTA attack success rates, exceeding baselines by an average of 52.1% and surpassing 90% on Claude-4-Sonnet
Source-stated
The authors report that ARMs-Bench contains over 30K red-teaming instances across 51 risk categories, and that safety fine-tuning with it improves robustness while preserving general utility.
comprising over 30K red-teaming instances spanning 51 diverse risk categories
Source-stated
The proposed approach aims to balance attack diversity and effectiveness through layered memory and epsilon-greedy exploration.
To balance the diversity and effectiveness of the attack, we design a layered memory with an epsilon-greedy attack exploration algorithm.
Source and contributor context
Chen, Zhaorun, Liu, Xun, Kang, Mintong, Zhang, Jiawei, Pan, Minzhou, Yang, Shuang, Li, Bo · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Red-teaming
Deliberately attacking or tricking a system to find its weaknesses before others do.
Multimodal model
A model that works with more than one kind of input or output, such as text, images and audio.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Fine-tuning
Further training of an existing model to adapt it to a particular task or use.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: e3cae356-876c-4194-808c-9e1d84ca87cf.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-08T04:32:30.460Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 4893ad348df38fba8e86e36b31b85cfed5e96b121cefa7615b1a16302d9dd771. Source text hash: 1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc.
passed · 2026-11-08T04:32:30.460Z · anthropic/claude-opus-5.5
Publication: e3cae356-876c-4194-808c-9e1d84ca87cf
attribution — pass: The reading attributes the paper to Zhaorun Chen and coauthors and keeps the performance and fine-tuning results framed as the authors' reports. This matches the arXiv author list and abstract.
Authors:Zhaorun Chen, Xun Liu, Mintong Kang, Jiawei Zhang, Minzhou Pan, Shuang Yang, Bo Li
summary — pass: The reading names the actor, spells out vision-language models (VLMs), and describes ARMs as an automated way to test whether VLMs can be pushed into harmful outputs. It covers the method, results, dataset and fine-tuning claim, and notes that the evidence comes from benchmarks. Epsilon-greedy is not explained, but it is secondary and could get a term tooltip. The title and question follow from the abstract.
we propose ARMs, an adaptive red-teaming agent that systematically conducts comprehensive risk assessments for VLMs
date — pass: The originalPublished date of 2025-10-03 matches the v1 submission date. It is not the later v2 revision date of 4 Nov 2026 or the 2026 feed date.
[Submitted on 3 Oct 2025 (v1), last revised 4 Nov 2026 (this version, v2)]
support — pass: Every claim is stated in the abstract: the 11 strategies, the 17 integrated algorithms, layered memory with epsilon-greedy exploration, the 52.1% average gain over baselines and over 90% on Claude-4-Sonnet, greater instance diversity, ARMs-Bench at over 30K instances across 51 categories, and the fine-tuning outcome. The limitations accurately note that the abstract lacks evaluation detail and evidence about deployed systems, without overreaching.
Safety fine-tuning with ARMs-Bench substantially improves the robustness of VLMs while preserving their general utility
media — pass: The selected media is the verified direct link to the arXiv abstract page for this paper, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Deliberately attacking or tricking a system to find its weaknesses before others do.
Definition referenceA model that works with more than one kind of input or output, such as text, images and audio.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeFurther training of an existing model to adapt it to a particular task or use.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceFurther training of an existing model to adapt it to a particular task or use.
Definition referenceFurther training of an existing model to adapt it to a particular task or use.
Definition referenceDeliberately attacking or tricking a system to find its weaknesses before others do.
Definition referenceA model that works with more than one kind of input or output, such as text, images and audio.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeFurther training of an existing model to adapt it to a particular task or use.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceFurther training of an existing model to adapt it to a particular task or use.
Definition referenceFurther training of an existing model to adapt it to a particular task or use.
Definition reference