{"id":"f0d0758d5a99e3c6a360413acc82c3d90c83df061ba3d4e72f2fbc7f21f7072d","title":"arXiv cs.AI paper stress-tests whether LLM safety judges are policy-invariant","argument":"Shihao Weng, Yang Feng and Xiaofei Xie argue that a safety judge should assess an agent’s behavior consistently when evaluation policies are rewritten without changing their meaning. They call this property policy invariance and test it through three principles: rubric-semantics invariance, rubric-threshold invariance and ambiguity-aware calibration. In tests using four agent-class judges and trajectories from ASSEBench and R-Judge, the authors report that judges reacted comparably to meaningful policy shifts and meaning-preserving rewrites, without distinguishing them. Such rewrites flipped up to 9.1% of verdicts above baseline jitter; 18–43% of observed flips occurred on cases the authors classify as unambiguous. The authors say the findings also held with a frontier-scale judge, a third benchmark and a second base policy. Their concern is that safety scores can partly reflect how an evaluator is prompted, rather than only what an agent did. They introduce the Policy Invariance Score and Judge Card reporting protocol, which they say expose a wide spread in judge reliability that accuracy-only leaderboards miss. The abstract describes a stress-test and reports its results; it does not establish how often these effects occur in deployed safety evaluations.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence provided is the paper’s abstract, which does not give detailed methods or results for individual judges, benchmarks or policy rewrites. The abstract reports stress-test results but does not establish how often the reported effects occur in deployed safety evaluations.","speaker":{"name":"Weng, Shihao, Feng, Yang, Xie, Xiaofei","roleAtSource":"Article author"},"source":{"title":"Are LLM Safety Judges Policy-Invariant? A Three-Principle Stress-Test","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2605.06161","publishedAt":"2026-05-07","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that trustworthy safety judges should be policy-invariant and set out three principles for testing that property.","excerpt":"any trustworthy safety judge must satisfy a basic property we call policy invariance, and we operationalize it as three testable principles","url":"https://arxiv.org/abs/2605.06161","relationship":"Source-stated"},{"text":"In the reported tests, judges responded similarly to meaningful policy shifts and meaning-preserving structural rewrites, and did not distinguish the two.","excerpt":"today's judges respond to meaningful normative shifts and to meaningless structural rewrites with comparable strength, and cannot tell the two apart.","url":"https://arxiv.org/abs/2605.06161","relationship":"Source-stated"},{"text":"The authors report that content-preserving rewrites flipped up to 9.1% of verdicts above baseline jitter, and that 18–43% of observed flips occurred on unambiguous cases.","excerpt":"Content-preserving policy rewrites flip up to 9.1% of verdicts above baseline jitter, and 18-43% of all observed flips occur on unambiguous cases under such rewrites","url":"https://arxiv.org/abs/2605.06161","relationship":"Source-stated"},{"text":"The authors say their findings persisted with a frontier-scale judge, a third benchmark and a second base policy.","excerpt":"These findings persist with a frontier-scale judge, a third benchmark, and a second base policy.","url":"https://arxiv.org/abs/2605.06161","relationship":"Source-stated"},{"text":"The authors say their Policy Invariance Score and Judge Card reporting protocol expose a spread in judge reliability that accuracy-only leaderboards do not show.","excerpt":"we contribute the Policy Invariance Score and the Judge Card reporting protocol, which expose an order-of-magnitude spread in judge reliability that is invisible to accuracy-only leaderboards.","url":"https://arxiv.org/abs/2605.06161","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2605.06161","checkedAt":"2026-11-13T23:40:22.514Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How do the authors’ stress-test results vary across the judges, benchmarks and policy rewrites they tested?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI paper stress-tests whether LLM safety judges are policy-invariant","summary":"Shihao Weng, Yang Feng and Xiaofei Xie argue that a safety judge should assess an agent’s behavior consistently when evaluation policies are rewritten without changing their meaning. They call this property policy invariance and test it through three principles: rubric-semantics invariance, rubric-threshold invariance and ambiguity-aware calibration. In tests using four agent-class judges and trajectories from ASSEBench and R-Judge, the authors report that judges reacted comparably to meaningful policy shifts and meaning-preserving rewrites, without distinguishing them. Such rewrites flipped up to 9.1% of verdicts above baseline jitter; 18–43% of observed flips occurred on cases the authors classify as unambiguous. The authors say the findings also held with a frontier-scale judge, a third benchmark and a second base policy. Their concern is that safety scores can partly reflect how an evaluator is prompted, rather than only what an agent did. They introduce the Policy Invariance Score and Judge Card reporting protocol, which they say expose a wide spread in judge reliability that accuracy-only leaderboards miss. The abstract describes a stress-test and reports its results; it does not establish how often these effects occur in deployed safety evaluations.","question":"How do the authors’ stress-test results vary across the judges, benchmarks and policy rewrites they tested?","claims":[{"text":"The authors argue that trustworthy safety judges should be policy-invariant and set out three principles for testing that property.","excerpt":"any trustworthy safety judge must satisfy a basic property we call policy invariance, and we operationalize it as three testable principles","support":"source-stated"},{"text":"In the reported tests, judges responded similarly to meaningful policy shifts and meaning-preserving structural rewrites, and did not distinguish the two.","excerpt":"today's judges respond to meaningful normative shifts and to meaningless structural rewrites with comparable strength, and cannot tell the two apart.","support":"source-stated"},{"text":"The authors report that content-preserving rewrites flipped up to 9.1% of verdicts above baseline jitter, and that 18–43% of observed flips occurred on unambiguous cases.","excerpt":"Content-preserving policy rewrites flip up to 9.1% of verdicts above baseline jitter, and 18-43% of all observed flips occur on unambiguous cases under such rewrites","support":"source-stated"},{"text":"The authors say their findings persisted with a frontier-scale judge, a third benchmark and a second base policy.","excerpt":"These findings persist with a frontier-scale judge, a third benchmark, and a second base policy.","support":"source-stated"},{"text":"The authors say their Policy Invariance Score and Judge Card reporting protocol expose a spread in judge reliability that accuracy-only leaderboards do not show.","excerpt":"we contribute the Policy Invariance Score and the Judge Card reporting protocol, which expose an order-of-magnitude spread in judge reliability that is invisible to accuracy-only leaderboards.","support":"source-stated"}],"limitations":["The evidence provided is the paper’s abstract, which does not give detailed methods or results for individual judges, benchmarks or policy rewrites.","The abstract reports stress-test results but does not establish how often the reported effects occur in deployed safety evaluations."],"mediaId":"47bfeb648115b0944ad550bd4548a6414d7e20aed84e09bce08b299868ac9cef","correctionReason":null,"id":"f0d0758d5a99e3c6a360413acc82c3d90c83df061ba3d4e72f2fbc7f21f7072d","publicationId":"34824a32-f0dc-4530-8d30-55f376577a9c","version":1,"sourceUrl":"https://arxiv.org/abs/2605.06161","sourceTitle":"Are LLM Safety Judges Policy-Invariant? A Three-Principle Stress-Test","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-05-07","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:07:04.170Z","createdAt":"2026-11-13T21:33:54.045Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"47bfeb648115b0944ad550bd4548a6414d7e20aed84e09bce08b299868ac9cef","kind":"article","url":"https://arxiv.org/abs/2605.06161","sourceUrl":"https://arxiv.org/abs/2605.06161","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-13T21:32:40.758Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2605.06161","checkedAt":"2026-11-13T23:40:22.514Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v7","mediaId":"47bfeb648115b0944ad550bd4548a6414d7e20aed84e09bce08b299868ac9cef","confidence":0.79,"provider":"typesafe","model":"jev-1.13.0","inputHash":"958eb55e1b053cbfe9887a036c3cbfcde004af93753ebb2121bff130db599fdc","decidedAt":"2026-11-13T21:32:42.860Z"},"provenance":{"textHash":"6b5306566b102f4d753fe302b9844912a9717d5f2cb2fcf76cb4b8a8a97aae22","observationId":"68c300435d830bea184615f2cf89a9313ac71dd4ead2cceaf516355f24b4fc07","retrievedAt":"2026-11-13T21:32:40.439Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"f875649e-e5bf-4c60-8d07-2d8a0ffc975a"},"reviewer":{"model":"gpt-6-luna","runId":"2967d106-16ba-4c3b-8fc7-8976833deacf"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-13T21:33:54.045Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Weng, Shihao","url":null,"kind":"unknown"},{"name":"Feng, Yang","url":null,"kind":"unknown"},{"name":"Xie, Xiaofei","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Weng, Shihao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Weng, Shihao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.06161","locator":"head/meta[name=author|citation_author]"},"assertionId":"2c47328779a0a093df5b20020e7e40bb4f0d33d4dd1116e041d1fa5d2605dc3d:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T23:40:22.514Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2c47328779a0a093df5b20020e7e40bb4f0d33d4dd1116e041d1fa5d2605dc3d","sourceUrl":"https://arxiv.org/abs/2605.06161","contentSha256":"01c077f8aed781242387b2a826aa022d38572ce70989ae360e7969c4c0dccc49","textHash":"6b5306566b102f4d753fe302b9844912a9717d5f2cb2fcf76cb4b8a8a97aae22","observedAt":"2026-11-13T23:40:22.514Z"}},{"actor":{"name":"Feng, Yang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Feng, Yang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.06161","locator":"head/meta[name=author|citation_author]"},"assertionId":"2c47328779a0a093df5b20020e7e40bb4f0d33d4dd1116e041d1fa5d2605dc3d:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T23:40:22.514Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2c47328779a0a093df5b20020e7e40bb4f0d33d4dd1116e041d1fa5d2605dc3d","sourceUrl":"https://arxiv.org/abs/2605.06161","contentSha256":"01c077f8aed781242387b2a826aa022d38572ce70989ae360e7969c4c0dccc49","textHash":"6b5306566b102f4d753fe302b9844912a9717d5f2cb2fcf76cb4b8a8a97aae22","observedAt":"2026-11-13T23:40:22.514Z"}},{"actor":{"name":"Xie, Xiaofei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Xie, Xiaofei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.06161","locator":"head/meta[name=author|citation_author]"},"assertionId":"2c47328779a0a093df5b20020e7e40bb4f0d33d4dd1116e041d1fa5d2605dc3d:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T23:40:22.514Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2c47328779a0a093df5b20020e7e40bb4f0d33d4dd1116e041d1fa5d2605dc3d","sourceUrl":"https://arxiv.org/abs/2605.06161","contentSha256":"01c077f8aed781242387b2a826aa022d38572ce70989ae360e7969c4c0dccc49","textHash":"6b5306566b102f4d753fe302b9844912a9717d5f2cb2fcf76cb4b8a8a97aae22","observedAt":"2026-11-13T23:40:22.514Z"}}],"participationCoverage":{"total":3,"returned":3,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2605.06161","snapshotId":"2c47328779a0a093df5b20020e7e40bb4f0d33d4dd1116e041d1fa5d2605dc3d","contentSha256":"01c077f8aed781242387b2a826aa022d38572ce70989ae360e7969c4c0dccc49","textHash":"6b5306566b102f4d753fe302b9844912a9717d5f2cb2fcf76cb4b8a8a97aae22","observedAt":"2026-11-13T23:40:22.514Z","basis":"publisher-page-metadata","sourceLinks":45,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/f0d0758d5a99e3c6a360413acc82c3d90c83df061ba3d4e72f2fbc7f21f7072d/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-paper-stress-tests-whether-llm-safety-judges-are-f0d0758d","formats":{"html":"/readings/arxiv-cs-ai-paper-stress-tests-whether-llm-safety-judges-are-f0d0758d","markdown":"/readings/arxiv-cs-ai-paper-stress-tests-whether-llm-safety-judges-are-f0d0758d.md","json":"/readings/arxiv-cs-ai-paper-stress-tests-whether-llm-safety-judges-are-f0d0758d.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"},{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"f0d0758d5a99e3c6a360413acc82c3d90c83df061ba3d4e72f2fbc7f21f7072d","revision":"34824a32-f0dc-4530-8d30-55f376577a9c","revisionKind":"publication-id","title":"arXiv cs.AI paper stress-tests whether LLM safety judges are policy-invariant","url":"https://escapemeridian.com/readings/arxiv-cs-ai-paper-stress-tests-whether-llm-safety-judges-are-f0d0758d"},"source":{"type":"source","id":"https://arxiv.org/abs/2605.06161","revision":"6b5306566b102f4d753fe302b9844912a9717d5f2cb2fcf76cb4b8a8a97aae22","revisionKind":"sha256-source-text","title":"Are LLM Safety Judges Policy-Invariant? A Three-Principle Stress-Test","url":"https://arxiv.org/abs/2605.06161"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"f0d0758d5a99e3c6a360413acc82c3d90c83df061ba3d4e72f2fbc7f21f7072d","title":"arXiv cs.AI paper stress-tests whether LLM safety judges are policy-invariant","argument":"Shihao Weng, Yang Feng and Xiaofei Xie argue that a safety judge should assess an agent’s behavior consistently when evaluation policies are rewritten without changing their meaning. They call this property policy invariance and test it through three principles: rubric-semantics invariance, rubric-threshold invariance and ambiguity-aware calibration. In tests using four agent-class judges and trajectories from ASSEBench and R-Judge, the authors report that judges reacted comparably to meaningful policy shifts and meaning-preserving rewrites, without distinguishing them. Such rewrites flipped up to 9.1% of verdicts above baseline jitter; 18–43% of observed flips occurred on cases the authors classify as unambiguous. The authors say the findings also held with a frontier-scale judge, a third benchmark and a second base policy. Their concern is that safety scores can partly reflect how an evaluator is prompted, rather than only what an agent did. They introduce the Policy Invariance Score and Judge Card reporting protocol, which they say expose a wide spread in judge reliability that accuracy-only leaderboards miss. The abstract describes a stress-test and reports its results; it does not establish how often these effects occur in deployed safety evaluations.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence provided is the paper’s abstract, which does not give detailed methods or results for individual judges, benchmarks or policy rewrites. The abstract reports stress-test results but does not establish how often the reported effects occur in deployed safety evaluations.","speaker":{"name":"Weng, Shihao, Feng, Yang, Xie, Xiaofei","roleAtSource":"Article author"},"source":{"title":"Are LLM Safety Judges Policy-Invariant? A Three-Principle Stress-Test","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2605.06161","publishedAt":"2026-05-07","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that trustworthy safety judges should be policy-invariant and set out three principles for testing that property.","excerpt":"any trustworthy safety judge must satisfy a basic property we call policy invariance, and we operationalize it as three testable principles","url":"https://arxiv.org/abs/2605.06161","relationship":"Source-stated"},{"text":"In the reported tests, judges responded similarly to meaningful policy shifts and meaning-preserving structural rewrites, and did not distinguish the two.","excerpt":"today's judges respond to meaningful normative shifts and to meaningless structural rewrites with comparable strength, and cannot tell the two apart.","url":"https://arxiv.org/abs/2605.06161","relationship":"Source-stated"},{"text":"The authors report that content-preserving rewrites flipped up to 9.1% of verdicts above baseline jitter, and that 18–43% of observed flips occurred on unambiguous cases.","excerpt":"Content-preserving policy rewrites flip up to 9.1% of verdicts above baseline jitter, and 18-43% of all observed flips occur on unambiguous cases under such rewrites","url":"https://arxiv.org/abs/2605.06161","relationship":"Source-stated"},{"text":"The authors say their findings persisted with a frontier-scale judge, a third benchmark and a second base policy.","excerpt":"These findings persist with a frontier-scale judge, a third benchmark, and a second base policy.","url":"https://arxiv.org/abs/2605.06161","relationship":"Source-stated"},{"text":"The authors say their Policy Invariance Score and Judge Card reporting protocol expose a spread in judge reliability that accuracy-only leaderboards do not show.","excerpt":"we contribute the Policy Invariance Score and the Judge Card reporting protocol, which expose an order-of-magnitude spread in judge reliability that is invisible to accuracy-only leaderboards.","url":"https://arxiv.org/abs/2605.06161","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2605.06161","checkedAt":"2026-11-13T23:40:22.514Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How do the authors’ stress-test results vary across the judges, benchmarks and policy rewrites they tested?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI paper stress-tests whether LLM safety judges are policy-invariant","summary":"Shihao Weng, Yang Feng and Xiaofei Xie argue that a safety judge should assess an agent’s behavior consistently when evaluation policies are rewritten without changing their meaning. They call this property policy invariance and test it through three principles: rubric-semantics invariance, rubric-threshold invariance and ambiguity-aware calibration. In tests using four agent-class judges and trajectories from ASSEBench and R-Judge, the authors report that judges reacted comparably to meaningful policy shifts and meaning-preserving rewrites, without distinguishing them. Such rewrites flipped up to 9.1% of verdicts above baseline jitter; 18–43% of observed flips occurred on cases the authors classify as unambiguous. The authors say the findings also held with a frontier-scale judge, a third benchmark and a second base policy. Their concern is that safety scores can partly reflect how an evaluator is prompted, rather than only what an agent did. They introduce the Policy Invariance Score and Judge Card reporting protocol, which they say expose a wide spread in judge reliability that accuracy-only leaderboards miss. The abstract describes a stress-test and reports its results; it does not establish how often these effects occur in deployed safety evaluations.","question":"How do the authors’ stress-test results vary across the judges, benchmarks and policy rewrites they tested?","claims":[{"text":"The authors argue that trustworthy safety judges should be policy-invariant and set out three principles for testing that property.","excerpt":"any trustworthy safety judge must satisfy a basic property we call policy invariance, and we operationalize it as three testable principles","support":"source-stated"},{"text":"In the reported tests, judges responded similarly to meaningful policy shifts and meaning-preserving structural rewrites, and did not distinguish the two.","excerpt":"today's judges respond to meaningful normative shifts and to meaningless structural rewrites with comparable strength, and cannot tell the two apart.","support":"source-stated"},{"text":"The authors report that content-preserving rewrites flipped up to 9.1% of verdicts above baseline jitter, and that 18–43% of observed flips occurred on unambiguous cases.","excerpt":"Content-preserving policy rewrites flip up to 9.1% of verdicts above baseline jitter, and 18-43% of all observed flips occur on unambiguous cases under such rewrites","support":"source-stated"},{"text":"The authors say their findings persisted with a frontier-scale judge, a third benchmark and a second base policy.","excerpt":"These findings persist with a frontier-scale judge, a third benchmark, and a second base policy.","support":"source-stated"},{"text":"The authors say their Policy Invariance Score and Judge Card reporting protocol expose a spread in judge reliability that accuracy-only leaderboards do not show.","excerpt":"we contribute the Policy Invariance Score and the Judge Card reporting protocol, which expose an order-of-magnitude spread in judge reliability that is invisible to accuracy-only leaderboards.","support":"source-stated"}],"limitations":["The evidence provided is the paper’s abstract, which does not give detailed methods or results for individual judges, benchmarks or policy rewrites.","The abstract reports stress-test results but does not establish how often the reported effects occur in deployed safety evaluations."],"mediaId":"47bfeb648115b0944ad550bd4548a6414d7e20aed84e09bce08b299868ac9cef","correctionReason":null,"id":"f0d0758d5a99e3c6a360413acc82c3d90c83df061ba3d4e72f2fbc7f21f7072d","publicationId":"34824a32-f0dc-4530-8d30-55f376577a9c","version":1,"sourceUrl":"https://arxiv.org/abs/2605.06161","sourceTitle":"Are LLM Safety Judges Policy-Invariant? A Three-Principle Stress-Test","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-05-07","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:07:04.170Z","createdAt":"2026-11-13T21:33:54.045Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"47bfeb648115b0944ad550bd4548a6414d7e20aed84e09bce08b299868ac9cef","kind":"article","url":"https://arxiv.org/abs/2605.06161","sourceUrl":"https://arxiv.org/abs/2605.06161","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-13T21:32:40.758Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2605.06161","checkedAt":"2026-11-13T23:40:22.514Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v7","mediaId":"47bfeb648115b0944ad550bd4548a6414d7e20aed84e09bce08b299868ac9cef","confidence":0.79,"provider":"typesafe","model":"jev-1.13.0","inputHash":"958eb55e1b053cbfe9887a036c3cbfcde004af93753ebb2121bff130db599fdc","decidedAt":"2026-11-13T21:32:42.860Z"},"provenance":{"textHash":"6b5306566b102f4d753fe302b9844912a9717d5f2cb2fcf76cb4b8a8a97aae22","observationId":"68c300435d830bea184615f2cf89a9313ac71dd4ead2cceaf516355f24b4fc07","retrievedAt":"2026-11-13T21:32:40.439Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"f875649e-e5bf-4c60-8d07-2d8a0ffc975a"},"reviewer":{"model":"gpt-6-luna","runId":"2967d106-16ba-4c3b-8fc7-8976833deacf"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-13T21:33:54.045Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Weng, Shihao","url":null,"kind":"unknown"},{"name":"Feng, Yang","url":null,"kind":"unknown"},{"name":"Xie, Xiaofei","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Weng, Shihao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Weng, Shihao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.06161","locator":"head/meta[name=author|citation_author]"},"assertionId":"2c47328779a0a093df5b20020e7e40bb4f0d33d4dd1116e041d1fa5d2605dc3d:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T23:40:22.514Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2c47328779a0a093df5b20020e7e40bb4f0d33d4dd1116e041d1fa5d2605dc3d","sourceUrl":"https://arxiv.org/abs/2605.06161","contentSha256":"01c077f8aed781242387b2a826aa022d38572ce70989ae360e7969c4c0dccc49","textHash":"6b5306566b102f4d753fe302b9844912a9717d5f2cb2fcf76cb4b8a8a97aae22","observedAt":"2026-11-13T23:40:22.514Z"}},{"actor":{"name":"Feng, Yang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Feng, Yang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.06161","locator":"head/meta[name=author|citation_author]"},"assertionId":"2c47328779a0a093df5b20020e7e40bb4f0d33d4dd1116e041d1fa5d2605dc3d:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T23:40:22.514Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2c47328779a0a093df5b20020e7e40bb4f0d33d4dd1116e041d1fa5d2605dc3d","sourceUrl":"https://arxiv.org/abs/2605.06161","contentSha256":"01c077f8aed781242387b2a826aa022d38572ce70989ae360e7969c4c0dccc49","textHash":"6b5306566b102f4d753fe302b9844912a9717d5f2cb2fcf76cb4b8a8a97aae22","observedAt":"2026-11-13T23:40:22.514Z"}},{"actor":{"name":"Xie, Xiaofei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Xie, Xiaofei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2605.06161","locator":"head/meta[name=author|citation_author]"},"assertionId":"2c47328779a0a093df5b20020e7e40bb4f0d33d4dd1116e041d1fa5d2605dc3d:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T23:40:22.514Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2c47328779a0a093df5b20020e7e40bb4f0d33d4dd1116e041d1fa5d2605dc3d","sourceUrl":"https://arxiv.org/abs/2605.06161","contentSha256":"01c077f8aed781242387b2a826aa022d38572ce70989ae360e7969c4c0dccc49","textHash":"6b5306566b102f4d753fe302b9844912a9717d5f2cb2fcf76cb4b8a8a97aae22","observedAt":"2026-11-13T23:40:22.514Z"}}],"participationCoverage":{"total":3,"returned":3,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2605.06161","snapshotId":"2c47328779a0a093df5b20020e7e40bb4f0d33d4dd1116e041d1fa5d2605dc3d","contentSha256":"01c077f8aed781242387b2a826aa022d38572ce70989ae360e7969c4c0dccc49","textHash":"6b5306566b102f4d753fe302b9844912a9717d5f2cb2fcf76cb4b8a8a97aae22","observedAt":"2026-11-13T23:40:22.514Z","basis":"publisher-page-metadata","sourceLinks":45,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/f0d0758d5a99e3c6a360413acc82c3d90c83df061ba3d4e72f2fbc7f21f7072d/source"}}}],"audits":[{"publicationId":"34824a32-f0dc-4530-8d30-55f376577a9c","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the paper to Shihao Weng, Yang Feng and Xiaofei Xie, matching the arXiv listing. It keeps all findings framed as the authors' arguments and reported results.","excerpt":"Authors:Shihao Weng, Yang Feng, Xiaofei Xie"},{"dimension":"summary","status":"uncertain","explanation":"The facts are faithful, but a general reader is left without key context. The reading never says the 'safety judge' is itself an LLM prompted to grade agent safety, which the source frames as the de facto evaluator. 'Agent-class judges', 'trajectories', 'baseline jitter' and 'frontier-scale' are unexplained. The three principles are only named; the threshold principle (judges should respond to intentional strict-to-lenient shifts) is not explained. 'Order-of-magnitude spread' is softened to 'wide spread'. Smallest repair: add tooltips or brief glosses for these terms and the LLM-as-a-Judge context, and restore 'order-of-magnitude'.","excerpt":"LLM-as-a-Judge pipelines have become the de facto evaluator for agent safety"},{"dimension":"date","status":"pass","explanation":"The displayed date of 2026-05-07 matches the v1 submission date. It correctly ignores the v2 revision of 5 Nov 2026 and the November feed date.","excerpt":"Submitted on 7 May 2026 (v1), last revised 5 Nov 2026 (this version, v2)"},{"dimension":"support","status":"pass","explanation":"Each claim is directly supported by a verbatim excerpt. The figures (9.1%, 18-43%), the robustness checks, and the concern that scores conflate agent behavior with evaluator prompting all match the abstract. The limitations correctly note that the abstract does not establish how often these effects occur in deployed evaluations. The question is a genuine open issue rather than a leading claim.","excerpt":"so safety scores produced by these judges partly conflate what the agent did with how the evaluator was prompted"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the original arXiv abstract page, which is the source itself.","excerpt":null}],"createdAt":"2026-11-13T21:33:54.045Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}