{"id":"6d1dbf3faf9f0cf48f803e2a92ebaaf1e5368532e7148a2c8d8484d01920f844","title":"arXiv paper reports ARMs attack success above 90% on Claude-4-Sonnet","argument":"In their paper, Zhaorun Chen and coauthors present ARMs, an agent for testing vision-language models (VLMs) against harmful outputs. Given a target behavior or risk, ARMs adapts red-teaming strategies and combines 17 algorithms with 11 proposed multimodal attack strategies. The authors say layered memory and an epsilon-greedy exploration algorithm help balance attack diversity and effectiveness. On instance- and policy-based benchmarks, they report attack success rates averaging 52.1% above baselines and exceeding 90% on Claude-4-Sonnet. They also say ARMs generated more diverse test instances, which they describe as revealing emerging vulnerabilities. Using those instances, the authors built ARMs-Bench, a dataset of over 30K examples across 51 risk categories. They report that safety fine-tuning with it improves VLM robustness while preserving general utility. The paper’s approach is to automate varied tests rather than rely on a narrow set of patterns or extensive manual engineering. The reported gains concern the paper’s benchmarks and fine-tuning experiments; the supplied abstract does not detail how those results translate to deployed systems.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The supplied abstract reports benchmark and fine-tuning results but does not provide the underlying evaluation details or explain how results translate to deployed systems. The authors describe ARMs-Bench fine-tuning as improving robustness while preserving general utility; the abstract does not specify the measures used for those outcomes.","speaker":{"name":"Chen, Zhaorun, Liu, Xun, Kang, Mintong, Zhang, Jiawei, Pan, Minzhou, Yang, Shuang, Li, Bo","roleAtSource":"Article author"},"source":{"title":"ARMs: Adaptive Red-Teaming Agent against Multimodal Models with Plug-and-Play Attacks","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2510.02677","publishedAt":"2025-10-03","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors propose ARMs, an agent that adapts red-teaming strategies to a specified harmful behavior or risk in vision-language models.","excerpt":"Given a target harmful behavior or risk definition, ARMs automatically optimizes diverse red-teaming strategies","url":"https://arxiv.org/abs/2510.02677","relationship":"Source-stated"},{"text":"The authors say ARMs combines 11 proposed multimodal attack strategies with 17 red-teaming algorithms.","excerpt":"We propose 11 novel multimodal attack strategies","url":"https://arxiv.org/abs/2510.02677","relationship":"Source-stated"},{"text":"On instance- and policy-based benchmarks, the authors report attack success rates averaging 52.1% above baselines and exceeding 90% on Claude-4-Sonnet.","excerpt":"achieves SOTA attack success rates, exceeding baselines by an average of 52.1% and surpassing 90% on Claude-4-Sonnet","url":"https://arxiv.org/abs/2510.02677","relationship":"Source-stated"},{"text":"The authors report that ARMs-Bench contains over 30K red-teaming instances across 51 risk categories, and that safety fine-tuning with it improves robustness while preserving general utility.","excerpt":"comprising over 30K red-teaming instances spanning 51 diverse risk categories","url":"https://arxiv.org/abs/2510.02677","relationship":"Source-stated"},{"text":"The proposed approach aims to balance attack diversity and effectiveness through layered memory and epsilon-greedy exploration.","excerpt":"To balance the diversity and effectiveness of the attack, we design a layered memory with an epsilon-greedy attack exploration algorithm.","url":"https://arxiv.org/abs/2510.02677","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2510.02677","checkedAt":"2026-11-08T04:48:06.696Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"What measures and results support the authors’ claim that safety fine-tuning with ARMs-Bench improves robustness while preserving general utility?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv paper reports ARMs attack success above 90% on Claude-4-Sonnet","summary":"In their paper, Zhaorun Chen and coauthors present ARMs, an agent for testing vision-language models (VLMs) against harmful outputs. Given a target behavior or risk, ARMs adapts red-teaming strategies and combines 17 algorithms with 11 proposed multimodal attack strategies. The authors say layered memory and an epsilon-greedy exploration algorithm help balance attack diversity and effectiveness. On instance- and policy-based benchmarks, they report attack success rates averaging 52.1% above baselines and exceeding 90% on Claude-4-Sonnet. They also say ARMs generated more diverse test instances, which they describe as revealing emerging vulnerabilities. Using those instances, the authors built ARMs-Bench, a dataset of over 30K examples across 51 risk categories. They report that safety fine-tuning with it improves VLM robustness while preserving general utility. The paper’s approach is to automate varied tests rather than rely on a narrow set of patterns or extensive manual engineering. The reported gains concern the paper’s benchmarks and fine-tuning experiments; the supplied abstract does not detail how those results translate to deployed systems.","question":"What measures and results support the authors’ claim that safety fine-tuning with ARMs-Bench improves robustness while preserving general utility?","claims":[{"text":"The authors propose ARMs, an agent that adapts red-teaming strategies to a specified harmful behavior or risk in vision-language models.","excerpt":"Given a target harmful behavior or risk definition, ARMs automatically optimizes diverse red-teaming strategies","support":"source-stated"},{"text":"The authors say ARMs combines 11 proposed multimodal attack strategies with 17 red-teaming algorithms.","excerpt":"We propose 11 novel multimodal attack strategies","support":"source-stated"},{"text":"On instance- and policy-based benchmarks, the authors report attack success rates averaging 52.1% above baselines and exceeding 90% on Claude-4-Sonnet.","excerpt":"achieves SOTA attack success rates, exceeding baselines by an average of 52.1% and surpassing 90% on Claude-4-Sonnet","support":"source-stated"},{"text":"The authors report that ARMs-Bench contains over 30K red-teaming instances across 51 risk categories, and that safety fine-tuning with it improves robustness while preserving general utility.","excerpt":"comprising over 30K red-teaming instances spanning 51 diverse risk categories","support":"source-stated"},{"text":"The proposed approach aims to balance attack diversity and effectiveness through layered memory and epsilon-greedy exploration.","excerpt":"To balance the diversity and effectiveness of the attack, we design a layered memory with an epsilon-greedy attack exploration algorithm.","support":"source-stated"}],"limitations":["The supplied abstract reports benchmark and fine-tuning results but does not provide the underlying evaluation details or explain how results translate to deployed systems.","The authors describe ARMs-Bench fine-tuning as improving robustness while preserving general utility; the abstract does not specify the measures used for those outcomes."],"mediaId":"3869b9d00896301132c1c3cc322c22392bf5546e2a36a25c94eb33c195b0d001","correctionReason":null,"id":"6d1dbf3faf9f0cf48f803e2a92ebaaf1e5368532e7148a2c8d8484d01920f844","publicationId":"e3cae356-876c-4194-808c-9e1d84ca87cf","version":1,"sourceUrl":"https://arxiv.org/abs/2510.02677","sourceTitle":"ARMs: Adaptive Red-Teaming Agent against Multimodal Models with Plug-and-Play Attacks","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2025-10-03","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:03:35.401Z","createdAt":"2026-11-08T04:32:30.460Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"3869b9d00896301132c1c3cc322c22392bf5546e2a36a25c94eb33c195b0d001","kind":"article","url":"https://arxiv.org/abs/2510.02677","sourceUrl":"https://arxiv.org/abs/2510.02677","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-08T04:31:40.463Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2510.02677","checkedAt":"2026-11-08T04:48:06.696Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"3869b9d00896301132c1c3cc322c22392bf5546e2a36a25c94eb33c195b0d001","confidence":0.87,"provider":"typesafe","model":"jev-1.13.0","inputHash":"7456a2f687a147479a487d0131e9d4dece5967c69e527e2f3108d641a1c9184b","decidedAt":"2026-11-08T04:31:42.760Z"},"provenance":{"textHash":"1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc","observationId":"4893ad348df38fba8e86e36b31b85cfed5e96b121cefa7615b1a16302d9dd771","retrievedAt":"2026-11-08T04:31:40.113Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"f7e7d432-5c91-40e7-8371-05f34723a5d1"},"reviewer":{"model":"gpt-6-luna","runId":"c3a220d4-929e-4569-8b9e-3acbfa99eed8"}},"audit":{"status":"passed","reviewedAt":"2026-11-08T04:32:30.460Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Chen, Zhaorun","url":null,"kind":"unknown"},{"name":"Liu, Xun","url":null,"kind":"unknown"},{"name":"Kang, Mintong","url":null,"kind":"unknown"},{"name":"Zhang, Jiawei","url":null,"kind":"unknown"},{"name":"Pan, Minzhou","url":null,"kind":"unknown"},{"name":"Yang, Shuang","url":null,"kind":"unknown"},{"name":"Li, Bo","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Chen, Zhaorun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chen, Zhaorun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2510.02677","locator":"head/meta[name=author|citation_author]"},"assertionId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:48:06.696Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba","sourceUrl":"https://arxiv.org/abs/2510.02677","contentSha256":"c4bd53287f93dc28033f18febf6533d1140c34dad329c34db9eae70627f98dc5","textHash":"1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc","observedAt":"2026-11-08T04:48:06.696Z"}},{"actor":{"name":"Liu, Xun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liu, Xun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2510.02677","locator":"head/meta[name=author|citation_author]"},"assertionId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:48:06.696Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba","sourceUrl":"https://arxiv.org/abs/2510.02677","contentSha256":"c4bd53287f93dc28033f18febf6533d1140c34dad329c34db9eae70627f98dc5","textHash":"1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc","observedAt":"2026-11-08T04:48:06.696Z"}},{"actor":{"name":"Kang, Mintong","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Kang, Mintong","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2510.02677","locator":"head/meta[name=author|citation_author]"},"assertionId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:48:06.696Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba","sourceUrl":"https://arxiv.org/abs/2510.02677","contentSha256":"c4bd53287f93dc28033f18febf6533d1140c34dad329c34db9eae70627f98dc5","textHash":"1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc","observedAt":"2026-11-08T04:48:06.696Z"}},{"actor":{"name":"Zhang, Jiawei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Jiawei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2510.02677","locator":"head/meta[name=author|citation_author]"},"assertionId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:48:06.696Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba","sourceUrl":"https://arxiv.org/abs/2510.02677","contentSha256":"c4bd53287f93dc28033f18febf6533d1140c34dad329c34db9eae70627f98dc5","textHash":"1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc","observedAt":"2026-11-08T04:48:06.696Z"}},{"actor":{"name":"Pan, Minzhou","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Pan, Minzhou","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2510.02677","locator":"head/meta[name=author|citation_author]"},"assertionId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:48:06.696Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba","sourceUrl":"https://arxiv.org/abs/2510.02677","contentSha256":"c4bd53287f93dc28033f18febf6533d1140c34dad329c34db9eae70627f98dc5","textHash":"1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc","observedAt":"2026-11-08T04:48:06.696Z"}},{"actor":{"name":"Yang, Shuang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yang, Shuang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2510.02677","locator":"head/meta[name=author|citation_author]"},"assertionId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:48:06.696Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba","sourceUrl":"https://arxiv.org/abs/2510.02677","contentSha256":"c4bd53287f93dc28033f18febf6533d1140c34dad329c34db9eae70627f98dc5","textHash":"1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc","observedAt":"2026-11-08T04:48:06.696Z"}},{"actor":{"name":"Li, Bo","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Bo","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2510.02677","locator":"head/meta[name=author|citation_author]"},"assertionId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:48:06.696Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba","sourceUrl":"https://arxiv.org/abs/2510.02677","contentSha256":"c4bd53287f93dc28033f18febf6533d1140c34dad329c34db9eae70627f98dc5","textHash":"1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc","observedAt":"2026-11-08T04:48:06.696Z"}}],"participationCoverage":{"total":7,"returned":7,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2510.02677","snapshotId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba","contentSha256":"c4bd53287f93dc28033f18febf6533d1140c34dad329c34db9eae70627f98dc5","textHash":"1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc","observedAt":"2026-11-08T04:48:06.696Z","basis":"publisher-page-metadata","sourceLinks":48,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/6d1dbf3faf9f0cf48f803e2a92ebaaf1e5368532e7148a2c8d8484d01920f844/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-paper-reports-arms-attack-success-above-90-on-claude-4-6d1dbf3f","formats":{"html":"/readings/arxiv-paper-reports-arms-attack-success-above-90-on-claude-4-6d1dbf3f","markdown":"/readings/arxiv-paper-reports-arms-attack-success-above-90-on-claude-4-6d1dbf3f.md","json":"/readings/arxiv-paper-reports-arms-attack-success-above-90-on-claude-4-6d1dbf3f.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-red-teaming","name":"Red-teaming","explanation":"Deliberately attacking or tricking a system to find its weaknesses before others do.","referenceUrl":"https://en.wikipedia.org/wiki/Red_team"},{"id":"term-multimodal-model","name":"Multimodal model","explanation":"A model that works with more than one kind of input or output, such as text, images and audio.","referenceUrl":"https://developers.google.com/machine-learning/glossary#multimodal-model"},{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-fine-tuning","name":"Fine-tuning","explanation":"Further training of an existing model to adapt it to a particular task or use.","referenceUrl":"https://developers.google.com/machine-learning/glossary#fine-tuning"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"},{"id":"claude","name":"Claude","kind":"model","context":"Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.","referenceUrls":["https://claude.com/product/overview","https://www.anthropic.com/news","https://www.anthropic.com/research/team/economics"],"path":"/entities/claude","json":"/entities/claude.json","markdown":"/entities/claude.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"6d1dbf3faf9f0cf48f803e2a92ebaaf1e5368532e7148a2c8d8484d01920f844","revision":"e3cae356-876c-4194-808c-9e1d84ca87cf","revisionKind":"publication-id","title":"arXiv paper reports ARMs attack success above 90% on Claude-4-Sonnet","url":"https://escapemeridian.com/readings/arxiv-paper-reports-arms-attack-success-above-90-on-claude-4-6d1dbf3f"},"source":{"type":"source","id":"https://arxiv.org/abs/2510.02677","revision":"1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc","revisionKind":"sha256-source-text","title":"ARMs: Adaptive Red-Teaming Agent against Multimodal Models with Plug-and-Play Attacks","url":"https://arxiv.org/abs/2510.02677"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"6d1dbf3faf9f0cf48f803e2a92ebaaf1e5368532e7148a2c8d8484d01920f844","title":"arXiv paper reports ARMs attack success above 90% on Claude-4-Sonnet","argument":"In their paper, Zhaorun Chen and coauthors present ARMs, an agent for testing vision-language models (VLMs) against harmful outputs. Given a target behavior or risk, ARMs adapts red-teaming strategies and combines 17 algorithms with 11 proposed multimodal attack strategies. The authors say layered memory and an epsilon-greedy exploration algorithm help balance attack diversity and effectiveness. On instance- and policy-based benchmarks, they report attack success rates averaging 52.1% above baselines and exceeding 90% on Claude-4-Sonnet. They also say ARMs generated more diverse test instances, which they describe as revealing emerging vulnerabilities. Using those instances, the authors built ARMs-Bench, a dataset of over 30K examples across 51 risk categories. They report that safety fine-tuning with it improves VLM robustness while preserving general utility. The paper’s approach is to automate varied tests rather than rely on a narrow set of patterns or extensive manual engineering. The reported gains concern the paper’s benchmarks and fine-tuning experiments; the supplied abstract does not detail how those results translate to deployed systems.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The supplied abstract reports benchmark and fine-tuning results but does not provide the underlying evaluation details or explain how results translate to deployed systems. The authors describe ARMs-Bench fine-tuning as improving robustness while preserving general utility; the abstract does not specify the measures used for those outcomes.","speaker":{"name":"Chen, Zhaorun, Liu, Xun, Kang, Mintong, Zhang, Jiawei, Pan, Minzhou, Yang, Shuang, Li, Bo","roleAtSource":"Article author"},"source":{"title":"ARMs: Adaptive Red-Teaming Agent against Multimodal Models with Plug-and-Play Attacks","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2510.02677","publishedAt":"2025-10-03","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors propose ARMs, an agent that adapts red-teaming strategies to a specified harmful behavior or risk in vision-language models.","excerpt":"Given a target harmful behavior or risk definition, ARMs automatically optimizes diverse red-teaming strategies","url":"https://arxiv.org/abs/2510.02677","relationship":"Source-stated"},{"text":"The authors say ARMs combines 11 proposed multimodal attack strategies with 17 red-teaming algorithms.","excerpt":"We propose 11 novel multimodal attack strategies","url":"https://arxiv.org/abs/2510.02677","relationship":"Source-stated"},{"text":"On instance- and policy-based benchmarks, the authors report attack success rates averaging 52.1% above baselines and exceeding 90% on Claude-4-Sonnet.","excerpt":"achieves SOTA attack success rates, exceeding baselines by an average of 52.1% and surpassing 90% on Claude-4-Sonnet","url":"https://arxiv.org/abs/2510.02677","relationship":"Source-stated"},{"text":"The authors report that ARMs-Bench contains over 30K red-teaming instances across 51 risk categories, and that safety fine-tuning with it improves robustness while preserving general utility.","excerpt":"comprising over 30K red-teaming instances spanning 51 diverse risk categories","url":"https://arxiv.org/abs/2510.02677","relationship":"Source-stated"},{"text":"The proposed approach aims to balance attack diversity and effectiveness through layered memory and epsilon-greedy exploration.","excerpt":"To balance the diversity and effectiveness of the attack, we design a layered memory with an epsilon-greedy attack exploration algorithm.","url":"https://arxiv.org/abs/2510.02677","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2510.02677","checkedAt":"2026-11-08T04:48:06.696Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"What measures and results support the authors’ claim that safety fine-tuning with ARMs-Bench improves robustness while preserving general utility?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv paper reports ARMs attack success above 90% on Claude-4-Sonnet","summary":"In their paper, Zhaorun Chen and coauthors present ARMs, an agent for testing vision-language models (VLMs) against harmful outputs. Given a target behavior or risk, ARMs adapts red-teaming strategies and combines 17 algorithms with 11 proposed multimodal attack strategies. The authors say layered memory and an epsilon-greedy exploration algorithm help balance attack diversity and effectiveness. On instance- and policy-based benchmarks, they report attack success rates averaging 52.1% above baselines and exceeding 90% on Claude-4-Sonnet. They also say ARMs generated more diverse test instances, which they describe as revealing emerging vulnerabilities. Using those instances, the authors built ARMs-Bench, a dataset of over 30K examples across 51 risk categories. They report that safety fine-tuning with it improves VLM robustness while preserving general utility. The paper’s approach is to automate varied tests rather than rely on a narrow set of patterns or extensive manual engineering. The reported gains concern the paper’s benchmarks and fine-tuning experiments; the supplied abstract does not detail how those results translate to deployed systems.","question":"What measures and results support the authors’ claim that safety fine-tuning with ARMs-Bench improves robustness while preserving general utility?","claims":[{"text":"The authors propose ARMs, an agent that adapts red-teaming strategies to a specified harmful behavior or risk in vision-language models.","excerpt":"Given a target harmful behavior or risk definition, ARMs automatically optimizes diverse red-teaming strategies","support":"source-stated"},{"text":"The authors say ARMs combines 11 proposed multimodal attack strategies with 17 red-teaming algorithms.","excerpt":"We propose 11 novel multimodal attack strategies","support":"source-stated"},{"text":"On instance- and policy-based benchmarks, the authors report attack success rates averaging 52.1% above baselines and exceeding 90% on Claude-4-Sonnet.","excerpt":"achieves SOTA attack success rates, exceeding baselines by an average of 52.1% and surpassing 90% on Claude-4-Sonnet","support":"source-stated"},{"text":"The authors report that ARMs-Bench contains over 30K red-teaming instances across 51 risk categories, and that safety fine-tuning with it improves robustness while preserving general utility.","excerpt":"comprising over 30K red-teaming instances spanning 51 diverse risk categories","support":"source-stated"},{"text":"The proposed approach aims to balance attack diversity and effectiveness through layered memory and epsilon-greedy exploration.","excerpt":"To balance the diversity and effectiveness of the attack, we design a layered memory with an epsilon-greedy attack exploration algorithm.","support":"source-stated"}],"limitations":["The supplied abstract reports benchmark and fine-tuning results but does not provide the underlying evaluation details or explain how results translate to deployed systems.","The authors describe ARMs-Bench fine-tuning as improving robustness while preserving general utility; the abstract does not specify the measures used for those outcomes."],"mediaId":"3869b9d00896301132c1c3cc322c22392bf5546e2a36a25c94eb33c195b0d001","correctionReason":null,"id":"6d1dbf3faf9f0cf48f803e2a92ebaaf1e5368532e7148a2c8d8484d01920f844","publicationId":"e3cae356-876c-4194-808c-9e1d84ca87cf","version":1,"sourceUrl":"https://arxiv.org/abs/2510.02677","sourceTitle":"ARMs: Adaptive Red-Teaming Agent against Multimodal Models with Plug-and-Play Attacks","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2025-10-03","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:03:35.401Z","createdAt":"2026-11-08T04:32:30.460Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"3869b9d00896301132c1c3cc322c22392bf5546e2a36a25c94eb33c195b0d001","kind":"article","url":"https://arxiv.org/abs/2510.02677","sourceUrl":"https://arxiv.org/abs/2510.02677","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-08T04:31:40.463Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2510.02677","checkedAt":"2026-11-08T04:48:06.696Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"3869b9d00896301132c1c3cc322c22392bf5546e2a36a25c94eb33c195b0d001","confidence":0.87,"provider":"typesafe","model":"jev-1.13.0","inputHash":"7456a2f687a147479a487d0131e9d4dece5967c69e527e2f3108d641a1c9184b","decidedAt":"2026-11-08T04:31:42.760Z"},"provenance":{"textHash":"1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc","observationId":"4893ad348df38fba8e86e36b31b85cfed5e96b121cefa7615b1a16302d9dd771","retrievedAt":"2026-11-08T04:31:40.113Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"f7e7d432-5c91-40e7-8371-05f34723a5d1"},"reviewer":{"model":"gpt-6-luna","runId":"c3a220d4-929e-4569-8b9e-3acbfa99eed8"}},"audit":{"status":"passed","reviewedAt":"2026-11-08T04:32:30.460Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Chen, Zhaorun","url":null,"kind":"unknown"},{"name":"Liu, Xun","url":null,"kind":"unknown"},{"name":"Kang, Mintong","url":null,"kind":"unknown"},{"name":"Zhang, Jiawei","url":null,"kind":"unknown"},{"name":"Pan, Minzhou","url":null,"kind":"unknown"},{"name":"Yang, Shuang","url":null,"kind":"unknown"},{"name":"Li, Bo","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Chen, Zhaorun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chen, Zhaorun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2510.02677","locator":"head/meta[name=author|citation_author]"},"assertionId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:48:06.696Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba","sourceUrl":"https://arxiv.org/abs/2510.02677","contentSha256":"c4bd53287f93dc28033f18febf6533d1140c34dad329c34db9eae70627f98dc5","textHash":"1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc","observedAt":"2026-11-08T04:48:06.696Z"}},{"actor":{"name":"Liu, Xun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liu, Xun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2510.02677","locator":"head/meta[name=author|citation_author]"},"assertionId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:48:06.696Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba","sourceUrl":"https://arxiv.org/abs/2510.02677","contentSha256":"c4bd53287f93dc28033f18febf6533d1140c34dad329c34db9eae70627f98dc5","textHash":"1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc","observedAt":"2026-11-08T04:48:06.696Z"}},{"actor":{"name":"Kang, Mintong","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Kang, Mintong","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2510.02677","locator":"head/meta[name=author|citation_author]"},"assertionId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:48:06.696Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba","sourceUrl":"https://arxiv.org/abs/2510.02677","contentSha256":"c4bd53287f93dc28033f18febf6533d1140c34dad329c34db9eae70627f98dc5","textHash":"1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc","observedAt":"2026-11-08T04:48:06.696Z"}},{"actor":{"name":"Zhang, Jiawei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Jiawei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2510.02677","locator":"head/meta[name=author|citation_author]"},"assertionId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:48:06.696Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba","sourceUrl":"https://arxiv.org/abs/2510.02677","contentSha256":"c4bd53287f93dc28033f18febf6533d1140c34dad329c34db9eae70627f98dc5","textHash":"1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc","observedAt":"2026-11-08T04:48:06.696Z"}},{"actor":{"name":"Pan, Minzhou","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Pan, Minzhou","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2510.02677","locator":"head/meta[name=author|citation_author]"},"assertionId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:48:06.696Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba","sourceUrl":"https://arxiv.org/abs/2510.02677","contentSha256":"c4bd53287f93dc28033f18febf6533d1140c34dad329c34db9eae70627f98dc5","textHash":"1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc","observedAt":"2026-11-08T04:48:06.696Z"}},{"actor":{"name":"Yang, Shuang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yang, Shuang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2510.02677","locator":"head/meta[name=author|citation_author]"},"assertionId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:48:06.696Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba","sourceUrl":"https://arxiv.org/abs/2510.02677","contentSha256":"c4bd53287f93dc28033f18febf6533d1140c34dad329c34db9eae70627f98dc5","textHash":"1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc","observedAt":"2026-11-08T04:48:06.696Z"}},{"actor":{"name":"Li, Bo","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Bo","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2510.02677","locator":"head/meta[name=author|citation_author]"},"assertionId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:48:06.696Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba","sourceUrl":"https://arxiv.org/abs/2510.02677","contentSha256":"c4bd53287f93dc28033f18febf6533d1140c34dad329c34db9eae70627f98dc5","textHash":"1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc","observedAt":"2026-11-08T04:48:06.696Z"}}],"participationCoverage":{"total":7,"returned":7,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2510.02677","snapshotId":"f2279dc4656abc37de3c19fa3d219e8f5846380c812b659b53c7ea350eb8cdba","contentSha256":"c4bd53287f93dc28033f18febf6533d1140c34dad329c34db9eae70627f98dc5","textHash":"1a941db52608a50daff6b84c43524ecfeea67492523b556143d1cd74ab083adc","observedAt":"2026-11-08T04:48:06.696Z","basis":"publisher-page-metadata","sourceLinks":48,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/6d1dbf3faf9f0cf48f803e2a92ebaaf1e5368532e7148a2c8d8484d01920f844/source"}}}],"audits":[{"publicationId":"e3cae356-876c-4194-808c-9e1d84ca87cf","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the paper to Zhaorun Chen and coauthors and keeps the performance and fine-tuning results framed as the authors' reports. This matches the arXiv author list and abstract.","excerpt":"Authors:Zhaorun Chen, Xun Liu, Mintong Kang, Jiawei Zhang, Minzhou Pan, Shuang Yang, Bo Li"},{"dimension":"summary","status":"pass","explanation":"The reading names the actor, spells out vision-language models (VLMs), and describes ARMs as an automated way to test whether VLMs can be pushed into harmful outputs. It covers the method, results, dataset and fine-tuning claim, and notes that the evidence comes from benchmarks. Epsilon-greedy is not explained, but it is secondary and could get a term tooltip. The title and question follow from the abstract.","excerpt":"we propose ARMs, an adaptive red-teaming agent that systematically conducts comprehensive risk assessments for VLMs"},{"dimension":"date","status":"pass","explanation":"The originalPublished date of 2025-10-03 matches the v1 submission date. It is not the later v2 revision date of 4 Nov 2026 or the 2026 feed date.","excerpt":"[Submitted on 3 Oct 2025 (v1), last revised 4 Nov 2026 (this version, v2)]"},{"dimension":"support","status":"pass","explanation":"Every claim is stated in the abstract: the 11 strategies, the 17 integrated algorithms, layered memory with epsilon-greedy exploration, the 52.1% average gain over baselines and over 90% on Claude-4-Sonnet, greater instance diversity, ARMs-Bench at over 30K instances across 51 categories, and the fine-tuning outcome. The limitations accurately note that the abstract lacks evaluation detail and evidence about deployed systems, without overreaching.","excerpt":"Safety fine-tuning with ARMs-Bench substantially improves the robustness of VLMs while preserving their general utility"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the arXiv abstract page for this paper, which is the most direct access to the source.","excerpt":null}],"createdAt":"2026-11-08T04:32:30.460Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}