{"id":"db48a40c0ec33d8e5ee2673cb8b21698f2e896a0d9ea079cb593ee2c94c892f5","title":"arXiv cs.AI: Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks","argument":"Youting Wang, Xiao Han, Dingyan Shang, Yuan Tang and Bowen Liu argue that agent-safety benchmark scores cannot be treated as interchangeable measures of an agent’s safety. They validate four benchmarks using their official implementations and scorers, and find that scoring choices and small model panels can distort comparisons. For example, on R-Judge, an “always positive” policy scores above five models that actually discriminate. The three broad-coverage benchmarks also rank the same models differently, while a reported trade-off between two measures changes with panel size. The paper’s central point is that validity depends on the outcome being measured: capability predicts task success but is negatively correlated with misalignment safety across 21 models. In a separate paired panel of 20 models, the contrast between these relationships is reported as significant. In an expanded panel, the misalignment correlation weakens, and neither change is significant. AgentHarm has the strongest reported held-out association with jailbreak safety, but both instruments score harmful compliance; the authors therefore describe this as evidence of convergent validity, not general safety. Their practical conclusion is that a safety claim should name the benchmark, metric, target behavior and model panel.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence supplied is the paper’s abstract, not its full methods or results; details needed to assess the benchmark implementations and analyses are not included here. The reported findings concern four benchmarks and specific model panels; the abstract does not establish how broadly they generalize to other benchmarks, models or target behaviors.","speaker":{"name":"Wang, Youting, Han, Xiao, Shang, Dingyan, Tang, Yuan, Liu, Bowen","roleAtSource":"Article author"},"source":{"title":"Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2607.28685","publishedAt":"2026-07-30","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that scores from different agent-safety benchmarks should not be treated as interchangeable measures of an agent’s safety.","excerpt":"their scores get quoted interchangeably as an agent's safety","url":"https://arxiv.org/abs/2607.28685","relationship":"Source-stated"},{"text":"On R-Judge, an “always positive” policy scores above five of the 21 models that actually discriminate.","excerpt":"on R-Judge that is $0.690$, above five of the 21 models that actually discriminate.","url":"https://arxiv.org/abs/2607.28685","relationship":"Source-stated"},{"text":"The three broad-coverage benchmarks rank the same 18 models differently, and the reported relationship between R-Judge specificity and AgentHarm safety changes with panel size.","excerpt":"The three broad-coverage benchmarks then rank the same 18 models differently","url":"https://arxiv.org/abs/2607.28685","relationship":"Source-stated"},{"text":"In the paired panel, capability predicts task success but correlates negatively with misalignment safety; the misalignment correlation weakens in the expanded panel, and neither change is significant.","excerpt":"On an expanded 41-model panel, the misalignment correlation weakens to $-0.16$ (95% CI $[-0.54, +0.22]$) and jailbreak strengthens to $+0.34$, though neither change is significant.","url":"https://arxiv.org/abs/2607.28685","relationship":"Source-stated"},{"text":"The authors say AgentHarm’s association with jailbreak safety is evidence of convergent validity rather than general safety, because both instruments score harmful compliance.","excerpt":"But both instruments score harmful compliance, so this is evidence of convergent validity rather than general safety.","url":"https://arxiv.org/abs/2607.28685","relationship":"Source-stated"},{"text":"The authors conclude that a safety claim should identify the benchmark, metric, target behavior and model panel.","excerpt":"Naming the benchmark, metric, target behavior, and model panel is the minimum a safety claim needs.","url":"https://arxiv.org/abs/2607.28685","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2607.28685","checkedAt":"2026-11-10T19:36:19.050Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well do these benchmark relationships hold for other model panels and target behaviors?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks","summary":"Youting Wang, Xiao Han, Dingyan Shang, Yuan Tang and Bowen Liu argue that agent-safety benchmark scores cannot be treated as interchangeable measures of an agent’s safety. They validate four benchmarks using their official implementations and scorers, and find that scoring choices and small model panels can distort comparisons. For example, on R-Judge, an “always positive” policy scores above five models that actually discriminate. The three broad-coverage benchmarks also rank the same models differently, while a reported trade-off between two measures changes with panel size. The paper’s central point is that validity depends on the outcome being measured: capability predicts task success but is negatively correlated with misalignment safety across 21 models. In a separate paired panel of 20 models, the contrast between these relationships is reported as significant. In an expanded panel, the misalignment correlation weakens, and neither change is significant. AgentHarm has the strongest reported held-out association with jailbreak safety, but both instruments score harmful compliance; the authors therefore describe this as evidence of convergent validity, not general safety. Their practical conclusion is that a safety claim should name the benchmark, metric, target behavior and model panel.","question":"How well do these benchmark relationships hold for other model panels and target behaviors?","claims":[{"text":"The authors argue that scores from different agent-safety benchmarks should not be treated as interchangeable measures of an agent’s safety.","excerpt":"their scores get quoted interchangeably as an agent's safety","support":"source-stated"},{"text":"On R-Judge, an “always positive” policy scores above five of the 21 models that actually discriminate.","excerpt":"on R-Judge that is $0.690$, above five of the 21 models that actually discriminate.","support":"source-stated"},{"text":"The three broad-coverage benchmarks rank the same 18 models differently, and the reported relationship between R-Judge specificity and AgentHarm safety changes with panel size.","excerpt":"The three broad-coverage benchmarks then rank the same 18 models differently","support":"source-stated"},{"text":"In the paired panel, capability predicts task success but correlates negatively with misalignment safety; the misalignment correlation weakens in the expanded panel, and neither change is significant.","excerpt":"On an expanded 41-model panel, the misalignment correlation weakens to $-0.16$ (95% CI $[-0.54, +0.22]$) and jailbreak strengthens to $+0.34$, though neither change is significant.","support":"source-stated"},{"text":"The authors say AgentHarm’s association with jailbreak safety is evidence of convergent validity rather than general safety, because both instruments score harmful compliance.","excerpt":"But both instruments score harmful compliance, so this is evidence of convergent validity rather than general safety.","support":"source-stated"},{"text":"The authors conclude that a safety claim should identify the benchmark, metric, target behavior and model panel.","excerpt":"Naming the benchmark, metric, target behavior, and model panel is the minimum a safety claim needs.","support":"source-stated"}],"limitations":["The evidence supplied is the paper’s abstract, not its full methods or results; details needed to assess the benchmark implementations and analyses are not included here.","The reported findings concern four benchmarks and specific model panels; the abstract does not establish how broadly they generalize to other benchmarks, models or target behaviors."],"mediaId":"360c6ea52c348edaa8e407a052ea9e6df98d9a5835796bcd8734da34668dd015","correctionReason":null,"id":"db48a40c0ec33d8e5ee2673cb8b21698f2e896a0d9ea079cb593ee2c94c892f5","publicationId":"4789de76-1119-46a4-8717-667cc7487964","version":1,"sourceUrl":"https://arxiv.org/abs/2607.28685","sourceTitle":"Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-07-30","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-05T05:00:27.965Z","createdAt":"2026-11-10T02:33:13.512Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"360c6ea52c348edaa8e407a052ea9e6df98d9a5835796bcd8734da34668dd015","kind":"article","url":"https://arxiv.org/abs/2607.28685","sourceUrl":"https://arxiv.org/abs/2607.28685","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-10T02:25:38.875Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2607.28685","checkedAt":"2026-11-10T19:36:19.050Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["useful_perspective"],"rubricVersion":"stream-admission-v5","mediaId":"360c6ea52c348edaa8e407a052ea9e6df98d9a5835796bcd8734da34668dd015","confidence":0.93,"provider":"typesafe","model":"jev-1.13.0","inputHash":"826d2feae2d8efedb2372efbe0255f259f686b8c532b7252b270117612910ead","decidedAt":"2026-11-10T02:25:40.998Z"},"provenance":{"textHash":"93bc4e005ccdd82ee9b648e741e3f64031663742167445897b61167c3650aacf","observationId":"1691b589dcbaecbfc851e1746b9e156d1c82c3b861a8e9f7bc75020fccadc80b","retrievedAt":"2026-11-10T02:25:36.188Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"07bd1d21-f4df-4652-8caf-dabf421a4480"},"reviewer":{"model":"gpt-6-luna","runId":"92fbeed6-f667-4e0c-8005-88aad1191ab6"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-10T02:33:13.512Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Wang, Youting","url":null,"kind":"unknown"},{"name":"Han, Xiao","url":null,"kind":"unknown"},{"name":"Shang, Dingyan","url":null,"kind":"unknown"},{"name":"Tang, Yuan","url":null,"kind":"unknown"},{"name":"Liu, Bowen","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Wang, Youting","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wang, Youting","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2607.28685","locator":"head/meta[name=author|citation_author]"},"assertionId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T19:36:19.050Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504","sourceUrl":"https://arxiv.org/abs/2607.28685","contentSha256":"c85facb91e70cc571a7d976879fcdcccd13c56f99525c6a77a4f8f697f7dea00","textHash":"93bc4e005ccdd82ee9b648e741e3f64031663742167445897b61167c3650aacf","observedAt":"2026-11-10T19:36:19.050Z"}},{"actor":{"name":"Han, Xiao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Han, Xiao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2607.28685","locator":"head/meta[name=author|citation_author]"},"assertionId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T19:36:19.050Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504","sourceUrl":"https://arxiv.org/abs/2607.28685","contentSha256":"c85facb91e70cc571a7d976879fcdcccd13c56f99525c6a77a4f8f697f7dea00","textHash":"93bc4e005ccdd82ee9b648e741e3f64031663742167445897b61167c3650aacf","observedAt":"2026-11-10T19:36:19.050Z"}},{"actor":{"name":"Shang, Dingyan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Shang, Dingyan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2607.28685","locator":"head/meta[name=author|citation_author]"},"assertionId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T19:36:19.050Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504","sourceUrl":"https://arxiv.org/abs/2607.28685","contentSha256":"c85facb91e70cc571a7d976879fcdcccd13c56f99525c6a77a4f8f697f7dea00","textHash":"93bc4e005ccdd82ee9b648e741e3f64031663742167445897b61167c3650aacf","observedAt":"2026-11-10T19:36:19.050Z"}},{"actor":{"name":"Tang, Yuan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Tang, Yuan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2607.28685","locator":"head/meta[name=author|citation_author]"},"assertionId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T19:36:19.050Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504","sourceUrl":"https://arxiv.org/abs/2607.28685","contentSha256":"c85facb91e70cc571a7d976879fcdcccd13c56f99525c6a77a4f8f697f7dea00","textHash":"93bc4e005ccdd82ee9b648e741e3f64031663742167445897b61167c3650aacf","observedAt":"2026-11-10T19:36:19.050Z"}},{"actor":{"name":"Liu, Bowen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liu, Bowen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2607.28685","locator":"head/meta[name=author|citation_author]"},"assertionId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T19:36:19.050Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504","sourceUrl":"https://arxiv.org/abs/2607.28685","contentSha256":"c85facb91e70cc571a7d976879fcdcccd13c56f99525c6a77a4f8f697f7dea00","textHash":"93bc4e005ccdd82ee9b648e741e3f64031663742167445897b61167c3650aacf","observedAt":"2026-11-10T19:36:19.050Z"}}],"participationCoverage":{"total":5,"returned":5,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2607.28685","snapshotId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504","contentSha256":"c85facb91e70cc571a7d976879fcdcccd13c56f99525c6a77a4f8f697f7dea00","textHash":"93bc4e005ccdd82ee9b648e741e3f64031663742167445897b61167c3650aacf","observedAt":"2026-11-10T19:36:19.050Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/db48a40c0ec33d8e5ee2673cb8b21698f2e896a0d9ea079cb593ee2c94c892f5/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-safety-or-just-capability-a-validity-audit-of-agent-db48a40c","formats":{"html":"/readings/arxiv-cs-ai-safety-or-just-capability-a-validity-audit-of-agent-db48a40c","markdown":"/readings/arxiv-cs-ai-safety-or-just-capability-a-validity-audit-of-agent-db48a40c.md","json":"/readings/arxiv-cs-ai-safety-or-just-capability-a-validity-audit-of-agent-db48a40c.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-ai-alignment","name":"AI alignment","explanation":"Research and practice aimed at making AI systems pursue the goals their builders and users intend.","referenceUrl":"https://en.wikipedia.org/wiki/AI_alignment"},{"id":"term-jailbreak","name":"Jailbreak","explanation":"A prompt written to trick an AI system into ignoring its safety rules.","referenceUrl":"https://arxiv.org/abs/2307.02483"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"db48a40c0ec33d8e5ee2673cb8b21698f2e896a0d9ea079cb593ee2c94c892f5","revision":"4789de76-1119-46a4-8717-667cc7487964","revisionKind":"publication-id","title":"arXiv cs.AI: Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks","url":"https://escapemeridian.com/readings/arxiv-cs-ai-safety-or-just-capability-a-validity-audit-of-agent-db48a40c"},"source":{"type":"source","id":"https://arxiv.org/abs/2607.28685","revision":"93bc4e005ccdd82ee9b648e741e3f64031663742167445897b61167c3650aacf","revisionKind":"sha256-source-text","title":"Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks","url":"https://arxiv.org/abs/2607.28685"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"db48a40c0ec33d8e5ee2673cb8b21698f2e896a0d9ea079cb593ee2c94c892f5","title":"arXiv cs.AI: Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks","argument":"Youting Wang, Xiao Han, Dingyan Shang, Yuan Tang and Bowen Liu argue that agent-safety benchmark scores cannot be treated as interchangeable measures of an agent’s safety. They validate four benchmarks using their official implementations and scorers, and find that scoring choices and small model panels can distort comparisons. For example, on R-Judge, an “always positive” policy scores above five models that actually discriminate. The three broad-coverage benchmarks also rank the same models differently, while a reported trade-off between two measures changes with panel size. The paper’s central point is that validity depends on the outcome being measured: capability predicts task success but is negatively correlated with misalignment safety across 21 models. In a separate paired panel of 20 models, the contrast between these relationships is reported as significant. In an expanded panel, the misalignment correlation weakens, and neither change is significant. AgentHarm has the strongest reported held-out association with jailbreak safety, but both instruments score harmful compliance; the authors therefore describe this as evidence of convergent validity, not general safety. Their practical conclusion is that a safety claim should name the benchmark, metric, target behavior and model panel.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence supplied is the paper’s abstract, not its full methods or results; details needed to assess the benchmark implementations and analyses are not included here. The reported findings concern four benchmarks and specific model panels; the abstract does not establish how broadly they generalize to other benchmarks, models or target behaviors.","speaker":{"name":"Wang, Youting, Han, Xiao, Shang, Dingyan, Tang, Yuan, Liu, Bowen","roleAtSource":"Article author"},"source":{"title":"Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2607.28685","publishedAt":"2026-07-30","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that scores from different agent-safety benchmarks should not be treated as interchangeable measures of an agent’s safety.","excerpt":"their scores get quoted interchangeably as an agent's safety","url":"https://arxiv.org/abs/2607.28685","relationship":"Source-stated"},{"text":"On R-Judge, an “always positive” policy scores above five of the 21 models that actually discriminate.","excerpt":"on R-Judge that is $0.690$, above five of the 21 models that actually discriminate.","url":"https://arxiv.org/abs/2607.28685","relationship":"Source-stated"},{"text":"The three broad-coverage benchmarks rank the same 18 models differently, and the reported relationship between R-Judge specificity and AgentHarm safety changes with panel size.","excerpt":"The three broad-coverage benchmarks then rank the same 18 models differently","url":"https://arxiv.org/abs/2607.28685","relationship":"Source-stated"},{"text":"In the paired panel, capability predicts task success but correlates negatively with misalignment safety; the misalignment correlation weakens in the expanded panel, and neither change is significant.","excerpt":"On an expanded 41-model panel, the misalignment correlation weakens to $-0.16$ (95% CI $[-0.54, +0.22]$) and jailbreak strengthens to $+0.34$, though neither change is significant.","url":"https://arxiv.org/abs/2607.28685","relationship":"Source-stated"},{"text":"The authors say AgentHarm’s association with jailbreak safety is evidence of convergent validity rather than general safety, because both instruments score harmful compliance.","excerpt":"But both instruments score harmful compliance, so this is evidence of convergent validity rather than general safety.","url":"https://arxiv.org/abs/2607.28685","relationship":"Source-stated"},{"text":"The authors conclude that a safety claim should identify the benchmark, metric, target behavior and model panel.","excerpt":"Naming the benchmark, metric, target behavior, and model panel is the minimum a safety claim needs.","url":"https://arxiv.org/abs/2607.28685","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2607.28685","checkedAt":"2026-11-10T19:36:19.050Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well do these benchmark relationships hold for other model panels and target behaviors?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks","summary":"Youting Wang, Xiao Han, Dingyan Shang, Yuan Tang and Bowen Liu argue that agent-safety benchmark scores cannot be treated as interchangeable measures of an agent’s safety. They validate four benchmarks using their official implementations and scorers, and find that scoring choices and small model panels can distort comparisons. For example, on R-Judge, an “always positive” policy scores above five models that actually discriminate. The three broad-coverage benchmarks also rank the same models differently, while a reported trade-off between two measures changes with panel size. The paper’s central point is that validity depends on the outcome being measured: capability predicts task success but is negatively correlated with misalignment safety across 21 models. In a separate paired panel of 20 models, the contrast between these relationships is reported as significant. In an expanded panel, the misalignment correlation weakens, and neither change is significant. AgentHarm has the strongest reported held-out association with jailbreak safety, but both instruments score harmful compliance; the authors therefore describe this as evidence of convergent validity, not general safety. Their practical conclusion is that a safety claim should name the benchmark, metric, target behavior and model panel.","question":"How well do these benchmark relationships hold for other model panels and target behaviors?","claims":[{"text":"The authors argue that scores from different agent-safety benchmarks should not be treated as interchangeable measures of an agent’s safety.","excerpt":"their scores get quoted interchangeably as an agent's safety","support":"source-stated"},{"text":"On R-Judge, an “always positive” policy scores above five of the 21 models that actually discriminate.","excerpt":"on R-Judge that is $0.690$, above five of the 21 models that actually discriminate.","support":"source-stated"},{"text":"The three broad-coverage benchmarks rank the same 18 models differently, and the reported relationship between R-Judge specificity and AgentHarm safety changes with panel size.","excerpt":"The three broad-coverage benchmarks then rank the same 18 models differently","support":"source-stated"},{"text":"In the paired panel, capability predicts task success but correlates negatively with misalignment safety; the misalignment correlation weakens in the expanded panel, and neither change is significant.","excerpt":"On an expanded 41-model panel, the misalignment correlation weakens to $-0.16$ (95% CI $[-0.54, +0.22]$) and jailbreak strengthens to $+0.34$, though neither change is significant.","support":"source-stated"},{"text":"The authors say AgentHarm’s association with jailbreak safety is evidence of convergent validity rather than general safety, because both instruments score harmful compliance.","excerpt":"But both instruments score harmful compliance, so this is evidence of convergent validity rather than general safety.","support":"source-stated"},{"text":"The authors conclude that a safety claim should identify the benchmark, metric, target behavior and model panel.","excerpt":"Naming the benchmark, metric, target behavior, and model panel is the minimum a safety claim needs.","support":"source-stated"}],"limitations":["The evidence supplied is the paper’s abstract, not its full methods or results; details needed to assess the benchmark implementations and analyses are not included here.","The reported findings concern four benchmarks and specific model panels; the abstract does not establish how broadly they generalize to other benchmarks, models or target behaviors."],"mediaId":"360c6ea52c348edaa8e407a052ea9e6df98d9a5835796bcd8734da34668dd015","correctionReason":null,"id":"db48a40c0ec33d8e5ee2673cb8b21698f2e896a0d9ea079cb593ee2c94c892f5","publicationId":"4789de76-1119-46a4-8717-667cc7487964","version":1,"sourceUrl":"https://arxiv.org/abs/2607.28685","sourceTitle":"Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-07-30","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-05T05:00:27.965Z","createdAt":"2026-11-10T02:33:13.512Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"360c6ea52c348edaa8e407a052ea9e6df98d9a5835796bcd8734da34668dd015","kind":"article","url":"https://arxiv.org/abs/2607.28685","sourceUrl":"https://arxiv.org/abs/2607.28685","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-10T02:25:38.875Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2607.28685","checkedAt":"2026-11-10T19:36:19.050Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["useful_perspective"],"rubricVersion":"stream-admission-v5","mediaId":"360c6ea52c348edaa8e407a052ea9e6df98d9a5835796bcd8734da34668dd015","confidence":0.93,"provider":"typesafe","model":"jev-1.13.0","inputHash":"826d2feae2d8efedb2372efbe0255f259f686b8c532b7252b270117612910ead","decidedAt":"2026-11-10T02:25:40.998Z"},"provenance":{"textHash":"93bc4e005ccdd82ee9b648e741e3f64031663742167445897b61167c3650aacf","observationId":"1691b589dcbaecbfc851e1746b9e156d1c82c3b861a8e9f7bc75020fccadc80b","retrievedAt":"2026-11-10T02:25:36.188Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"07bd1d21-f4df-4652-8caf-dabf421a4480"},"reviewer":{"model":"gpt-6-luna","runId":"92fbeed6-f667-4e0c-8005-88aad1191ab6"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-10T02:33:13.512Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Wang, Youting","url":null,"kind":"unknown"},{"name":"Han, Xiao","url":null,"kind":"unknown"},{"name":"Shang, Dingyan","url":null,"kind":"unknown"},{"name":"Tang, Yuan","url":null,"kind":"unknown"},{"name":"Liu, Bowen","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Wang, Youting","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wang, Youting","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2607.28685","locator":"head/meta[name=author|citation_author]"},"assertionId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T19:36:19.050Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504","sourceUrl":"https://arxiv.org/abs/2607.28685","contentSha256":"c85facb91e70cc571a7d976879fcdcccd13c56f99525c6a77a4f8f697f7dea00","textHash":"93bc4e005ccdd82ee9b648e741e3f64031663742167445897b61167c3650aacf","observedAt":"2026-11-10T19:36:19.050Z"}},{"actor":{"name":"Han, Xiao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Han, Xiao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2607.28685","locator":"head/meta[name=author|citation_author]"},"assertionId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T19:36:19.050Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504","sourceUrl":"https://arxiv.org/abs/2607.28685","contentSha256":"c85facb91e70cc571a7d976879fcdcccd13c56f99525c6a77a4f8f697f7dea00","textHash":"93bc4e005ccdd82ee9b648e741e3f64031663742167445897b61167c3650aacf","observedAt":"2026-11-10T19:36:19.050Z"}},{"actor":{"name":"Shang, Dingyan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Shang, Dingyan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2607.28685","locator":"head/meta[name=author|citation_author]"},"assertionId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T19:36:19.050Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504","sourceUrl":"https://arxiv.org/abs/2607.28685","contentSha256":"c85facb91e70cc571a7d976879fcdcccd13c56f99525c6a77a4f8f697f7dea00","textHash":"93bc4e005ccdd82ee9b648e741e3f64031663742167445897b61167c3650aacf","observedAt":"2026-11-10T19:36:19.050Z"}},{"actor":{"name":"Tang, Yuan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Tang, Yuan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2607.28685","locator":"head/meta[name=author|citation_author]"},"assertionId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T19:36:19.050Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504","sourceUrl":"https://arxiv.org/abs/2607.28685","contentSha256":"c85facb91e70cc571a7d976879fcdcccd13c56f99525c6a77a4f8f697f7dea00","textHash":"93bc4e005ccdd82ee9b648e741e3f64031663742167445897b61167c3650aacf","observedAt":"2026-11-10T19:36:19.050Z"}},{"actor":{"name":"Liu, Bowen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liu, Bowen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2607.28685","locator":"head/meta[name=author|citation_author]"},"assertionId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T19:36:19.050Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504","sourceUrl":"https://arxiv.org/abs/2607.28685","contentSha256":"c85facb91e70cc571a7d976879fcdcccd13c56f99525c6a77a4f8f697f7dea00","textHash":"93bc4e005ccdd82ee9b648e741e3f64031663742167445897b61167c3650aacf","observedAt":"2026-11-10T19:36:19.050Z"}}],"participationCoverage":{"total":5,"returned":5,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2607.28685","snapshotId":"8357299915dd3e51942cc3b7a796e1f7d7ea96c824f89422ebf227481bfea504","contentSha256":"c85facb91e70cc571a7d976879fcdcccd13c56f99525c6a77a4f8f697f7dea00","textHash":"93bc4e005ccdd82ee9b648e741e3f64031663742167445897b61167c3650aacf","observedAt":"2026-11-10T19:36:19.050Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/db48a40c0ec33d8e5ee2673cb8b21698f2e896a0d9ea079cb593ee2c94c892f5/source"}}}],"audits":[{"publicationId":"4789de76-1119-46a4-8717-667cc7487964","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The draft names the five listed authors (Wang, Han, Shang, Tang, Liu), attributes the arguments to them, and presents arXiv cs.AI as the venue. Findings are framed as reported or argued, not as established fact.","excerpt":"Authors:Youting Wang, Xiao Han, Dingyan Shang, Yuan Tang, Bowen Liu"},{"dimension":"summary","status":"uncertain","explanation":"Jargon blocks a non-specialist reader: agent, discriminate, specificity, held-out, misalignment safety, harmful compliance and convergent validity go unexplained. The always-positive example lacks its point: a trivial policy beats real models because of the F1 metric. The draft also blunts a key argument. It says the trade-off changes with panel size, but the authors call it a small-panel artifact. 'Neither change is significant' dangles because the jailbreak +0.34 shift is omitted. Calling outcome-dependence the paper's central point is EM framing. Smallest repairs: add tooltips or brief glosses, use 'artifact', and restore the jailbreak change.","excerpt":"the trade-off behind that disagreement is a small-panel artifact"},{"dimension":"date","status":"pass","explanation":"originalPublished 2026-07-30 matches the v1 submission date. It is not the 4 Nov 2026 v2 revision or the November feed date.","excerpt":"[Submitted on 30 Jul 2026 (v1), last revised 4 Nov 2026 (this version, v2)]"},{"dimension":"support","status":"uncertain","explanation":"Claim 4 places the capability correlations 'in the paired panel'. The source reports them at n=21 and reports only the contrast on the paired n=20 panel, so the scope should be corrected. Claim 3's excerpt supports only the ranking disagreement, not the panel-size change; the -0.64 at n=7 versus +0.02 at n=18 excerpt should be added. The other claims match their excerpts, and the limitations correctly note that only the abstract is available.","excerpt":"correlates negatively with misalignment safety ($\\rho{=}{-}0.44$, $n{=}21$). On their paired $n{=}20$ panel"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.","excerpt":null}],"createdAt":"2026-11-10T02:33:13.512Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}