Rhodes, Anthony, Singer, Gadi · Nov 6, 2026
arXiv cs.AI paper introduces BMMS for measuring policy compliance
Markdown · JSON · Original source
Anthony Rhodes and Gadi Singer introduce the Behavioral Metric Suite (BMMS), which evaluates whether a language model follows a particular policy corpus rather than scoring only preset behaviors such as or toxicity. Its measures include the Compliance Attainment Score (CAS), for severity-weighted attainment of binding obligations, and the Compliance Robustness Margin (CRM), for how much attainment survives adversarial pressure. The authors say a measure should be admitted only if its changes reflect the system being assessed, not the judging instrument. Across 1,003 obligations from eleven documents in five registers, the criterion reached 94.4% precision against blinded human judgment on vendor specifications, but 50.0% on analytical guidance. Re-scoring with judges from two other providers preserved target ordering, though one target’s absolute score varied. Comparing two detailed behavioral specifications, the authors report no resolved own-specification advantage after adjusting for document difficulty. They separately report that GPT-4o failed 70.7% of hard constraints in ’s Model Spec, while the two models failed 0.0% and 9.1% of their respective constraints. The results distinguish relative advantage from absolute compliance, but the reported comparisons cover a limited set of documents and systems.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 6, 2026 (2026-11-06).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-12T22:33:38.361Z. First observed: 2026-11-09T05:04:28.621Z. Retrieved: 2026-11-12T22:32:44.451Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
Some references could not be checked. Reload this page to try again.
Limits
The abstract reports tests across eleven documents and comparisons involving the two most detailed published behavioral specifications; it does not establish how the results generalize beyond those materials. The reported precision differs by register: 94.4% on vendor specifications and 50.0% on analytical guidance, so the criterion’s performance is not uniform across these categories. Re-scoring preserved target ordering, but the absolute score for a single target varied by up to 0.22 across judges from other providers. The abstract reports model-specific constraint failure rates; these figures alone do not establish performance in real-world use.
Open question
How would BMMS’s admission criterion and compliance comparisons perform on policy documents and systems beyond the two detailed specifications evaluated?
Source-stated
Rhodes and Singer introduce BMMS as a way to measure a language model’s compliance with a supplied policy corpus.
We introduce the Behavioral Misalignment Metric Suite (BMMS), which measures policy-conditioned misalignment by taking a policy corpus as an argument
Source-stated
The authors define an admission test intended to distinguish changes in the system being assessed from changes caused by the judging instrument.
a quantity that moves when the judge changes, holding the target fixed, is measuring the instrument.
Source-stated
Across 1,003 obligations from eleven documents in five registers, the criterion reached 94.4% precision on vendor specifications and 50.0% on analytical guidance, against blinded human judgment.
Across 1,003 obligations from eleven documents in five registers, the admission criterion holds 94.4% precision against blinded human judgment on vendor specifications and 50.0% on analytical guidance
Source-stated
The authors report that own-specification advantage was not resolved from zero after adjusting for document difficulty, while absolute self-compliance differed across the systems assessed.
Own-specification advantage is not resolved from zero once document difficulty is subtracted, but absolute self-compliance is another matter
Source-stated
In the reported Model Spec comparison, GPT-4o failed 70.7% of its binding hard constraints; the two Claude models failed 0.0% and 9.1% of theirs.
GPT-4o fails 70.7% of the hard constraints binding it in the Model Spec published by OpenAI, where the two Claude models fail 0.0% and 9.1% of theirs.
Source and contributor context
Rhodes, Anthony, Singer, Gadi · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
AI alignment
Research and practice aimed at making AI systems pursue the goals their builders and users intend.
Hallucination
When an AI system states something false or made up as if it were true.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
OpenAI
OpenAI develops AI models and products and publishes research, release announcements and evaluations.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: 2d3a140e-f4ad-4e92-80b0-1a9e628c4bc3.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T22:33:38.361Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 9e33461ee1485c753c5625e257a26aa5523cb3391c80d28376b4c46e79bb378b. Source text hash: 6c5e98bf3163206a357e76efcf1ded12030b07e031d59bd304ccb5be750425a4.
Revision and source-check history
Version 1 · 2026-11-12T22:33:38.361Z
arXiv cs.AI paper introduces BMMS for measuring policy compliance
Anthony Rhodes and Gadi Singer introduce the Behavioral Metric Suite (BMMS), which evaluates whether a language model follows a particular policy corpus rather than scoring only preset behaviors such as or toxicity. Its measures include the Compliance Attainment Score (CAS), for severity-weighted attainment of binding obligations, and the Compliance Robustness Margin (CRM), for how much attainment survives adversarial pressure. The authors say a measure should be admitted only if its changes reflect the system being assessed, not the judging instrument. Across 1,003 obligations from eleven documents in five registers, the criterion reached 94.4% precision against blinded human judgment on vendor specifications, but 50.0% on analytical guidance. Re-scoring with judges from two other providers preserved target ordering, though one target’s absolute score varied. Comparing two detailed behavioral specifications, the authors report no resolved own-specification advantage after adjusting for document difficulty. They separately report that GPT-4o failed 70.7% of hard constraints in ’s Model Spec, while the two models failed 0.0% and 9.1% of their respective constraints. The results distinguish relative advantage from absolute compliance, but the reported comparisons cover a limited set of documents and systems.
Original source
cs.AI · Article
Source details
Original source date: Nov 6, 2026 (2026-11-06).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-12T22:33:38.361Z. First observed: 2026-11-09T05:04:28.621Z. Retrieved: 2026-11-12T22:32:44.451Z.
Limits
The abstract reports tests across eleven documents and comparisons involving the two most detailed published behavioral specifications; it does not establish how the results generalize beyond those materials. The reported precision differs by register: 94.4% on vendor specifications and 50.0% on analytical guidance, so the criterion’s performance is not uniform across these categories. Re-scoring preserved target ordering, but the absolute score for a single target varied by up to 0.22 across judges from other providers. The abstract reports model-specific constraint failure rates; these figures alone do not establish performance in real-world use.
Open question
How would BMMS’s admission criterion and compliance comparisons perform on policy documents and systems beyond the two detailed specifications evaluated?
Source-stated
Rhodes and Singer introduce BMMS as a way to measure a language model’s compliance with a supplied policy corpus.
We introduce the Behavioral Misalignment Metric Suite (BMMS), which measures policy-conditioned misalignment by taking a policy corpus as an argument
Source-stated
The authors define an admission test intended to distinguish changes in the system being assessed from changes caused by the judging instrument.
a quantity that moves when the judge changes, holding the target fixed, is measuring the instrument.
Source-stated
Across 1,003 obligations from eleven documents in five registers, the criterion reached 94.4% precision on vendor specifications and 50.0% on analytical guidance, against blinded human judgment.
Across 1,003 obligations from eleven documents in five registers, the admission criterion holds 94.4% precision against blinded human judgment on vendor specifications and 50.0% on analytical guidance
Source-stated
The authors report that own-specification advantage was not resolved from zero after adjusting for document difficulty, while absolute self-compliance differed across the systems assessed.
Own-specification advantage is not resolved from zero once document difficulty is subtracted, but absolute self-compliance is another matter
Source-stated
In the reported Model Spec comparison, GPT-4o failed 70.7% of its binding hard constraints; the two Claude models failed 0.0% and 9.1% of theirs.
GPT-4o fails 70.7% of the hard constraints binding it in the Model Spec published by OpenAI, where the two Claude models fail 0.0% and 9.1% of theirs.
Source and contributor context
Rhodes, Anthony, Singer, Gadi · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
AI alignment
Research and practice aimed at making AI systems pursue the goals their builders and users intend.
Hallucination
When an AI system states something false or made up as if it were true.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
OpenAI
OpenAI develops AI models and products and publishes research, release announcements and evaluations.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: 2d3a140e-f4ad-4e92-80b0-1a9e628c4bc3.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T22:33:38.361Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 9e33461ee1485c753c5625e257a26aa5523cb3391c80d28376b4c46e79bb378b. Source text hash: 6c5e98bf3163206a357e76efcf1ded12030b07e031d59bd304ccb5be750425a4.
uncertain · 2026-11-12T22:33:38.361Z · anthropic/claude-opus-5.5
Publication: 2d3a140e-f4ad-4e92-80b0-1a9e628c4bc3
attribution — pass: The draft correctly attributes BMMS, its metrics, and all reported results to authors Anthony Rhodes and Gadi Singer. It frames the findings as the authors' reports, not as established fact.
Authors:Anthony Rhodes, Gadi Singer
summary — uncertain: The summary is faithful but hard for a non-specialist to follow. 'Own-specification advantage' is central and never explained; a minimal repair is to say it means whether a model scores better on its own developer's spec. 'Five registers' and 'precision against blinded human judgment' are also unexplained. The draft also leaves out a pointed detail: applying the admission test excluded one of the suite's own reported quantities. One short added clause would restore that.
Applying it excluded one of the quantities the suite itself reported.
date — pass: The originalPublished date of 2026-11-06 matches the v1 submission date on arXiv. The later feed date is correctly not used.
[Submitted on 6 Nov 2026]
support — pass: The claims, numbers and limitations match the abstract: 1,003 obligations, 94.4% versus 50.0% precision, the 0.22 score spread, and the 70.7%, 0.0% and 9.1% failure rates. The limitations and the question stay within the evidence and do not turn the test scores into real-world reliability.
re-scoring under judges from two other providers leaves target ordering invariant while the absolute score of a single target spans up to 0.22
media — pass: The selected media is the verified arXiv abstract page, which is the direct original source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Research and practice aimed at making AI systems pursue the goals their builders and users intend.
Definition referenceWhen an AI system states something false or made up as if it were true.
Definition referenceOpenAI develops AI models and products and publishes research, release announcements and evaluations.
Explore OpenAIClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivResearch and practice aimed at making AI systems pursue the goals their builders and users intend.
Definition referenceWhen an AI system states something false or made up as if it were true.
Definition referenceOpenAI develops AI models and products and publishes research, release announcements and evaluations.
Explore OpenAIClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv