# arXiv cs.AI paper introduces BMMS for measuring policy compliance [Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-paper-introduces-bmms-for-measuring-policy-78745fe1) Anthony Rhodes and Gadi Singer introduce the Behavioral Misalignment Metric Suite (BMMS), which evaluates whether a language model follows a particular policy corpus rather than scoring only preset behaviors such as hallucination or toxicity. Its measures include the Compliance Attainment Score (CAS), for severity-weighted attainment of binding obligations, and the Compliance Robustness Margin (CRM), for how much attainment survives adversarial pressure. The authors say a measure should be admitted only if its changes reflect the system being assessed, not the judging instrument. Across 1,003 obligations from eleven documents in five registers, the criterion reached 94.4% precision against blinded human judgment on vendor specifications, but 50.0% on analytical guidance. Re-scoring with judges from two other providers preserved target ordering, though one target’s absolute score varied. Comparing two detailed behavioral specifications, the authors report no resolved own-specification advantage after adjusting for document difficulty. They separately report that GPT-4o failed 70.7% of hard constraints in OpenAI’s Model Spec, while the two Claude models failed 0.0% and 9.1% of their respective constraints. The results distinguish relative advantage from absolute compliance, but the reported comparisons cover a limited set of documents and systems. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Nov 6, 2026 (2026-11-06). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-12T22:33:38.361Z. First observed: 2026-11-09T05:04:28.621Z. Retrieved: 2026-11-12T22:32:44.451Z. - [BMMS: Policy-Conditioned Measurement of Behavioral Compliance in Language Models](https://arxiv.org/abs/2611.09147) ## Limits The abstract reports tests across eleven documents and comparisons involving the two most detailed published behavioral specifications; it does not establish how the results generalize beyond those materials. The reported precision differs by register: 94.4% on vendor specifications and 50.0% on analytical guidance, so the criterion’s performance is not uniform across these categories. Re-scoring preserved target ordering, but the absolute score for a single target varied by up to 0.22 across judges from other providers. The abstract reports model-specific constraint failure rates; these figures alone do not establish performance in real-world use. ## Open question How would BMMS’s admission criterion and compliance comparisons perform on policy documents and systems beyond the two detailed specifications evaluated? ## Source-stated Rhodes and Singer introduce BMMS as a way to measure a language model’s compliance with a supplied policy corpus. > We introduce the Behavioral Misalignment Metric Suite (BMMS), which measures policy-conditioned misalignment by taking a policy corpus as an argument - [Supporting source](https://arxiv.org/abs/2611.09147) ## Source-stated The authors define an admission test intended to distinguish changes in the system being assessed from changes caused by the judging instrument. > a quantity that moves when the judge changes, holding the target fixed, is measuring the instrument. - [Supporting source](https://arxiv.org/abs/2611.09147) ## Source-stated Across 1,003 obligations from eleven documents in five registers, the criterion reached 94.4% precision on vendor specifications and 50.0% on analytical guidance, against blinded human judgment. > Across 1,003 obligations from eleven documents in five registers, the admission criterion holds 94.4% precision against blinded human judgment on vendor specifications and 50.0% on analytical guidance - [Supporting source](https://arxiv.org/abs/2611.09147) ## Source-stated The authors report that own-specification advantage was not resolved from zero after adjusting for document difficulty, while absolute self-compliance differed across the systems assessed. > Own-specification advantage is not resolved from zero once document difficulty is subtracted, but absolute self-compliance is another matter - [Supporting source](https://arxiv.org/abs/2611.09147) ## Source-stated In the reported Model Spec comparison, GPT-4o failed 70.7% of its binding hard constraints; the two Claude models failed 0.0% and 9.1% of theirs. > GPT-4o fails 70.7% of the hard constraints binding it in the Model Spec published by OpenAI, where the two Claude models fail 0.0% and 9.1% of theirs. - [Supporting source](https://arxiv.org/abs/2611.09147) ## Source and contributor context Rhodes, Anthony, Singer, Gadi · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## AI alignment Research and practice aimed at making AI systems pursue the goals their builders and users intend. - [Definition reference](https://en.wikipedia.org/wiki/AI_alignment) ## Hallucination When an AI system states something false or made up as if it were true. - [Definition reference](https://developers.google.com/machine-learning/glossary#hallucination) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## OpenAI OpenAI develops AI models and products and publishes research, release announcements and evaluations. - [Explore this context](/entities/openai) - [Reference](https://openai.com/news/) ## Claude Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding. - [Explore this context](/entities/claude) - [Reference](https://claude.com/product/overview) - [Reference](https://www.anthropic.com/news) - [Reference](https://www.anthropic.com/research/team/economics) ## Editorial provenance Version 1. Publication ID: 2d3a140e-f4ad-4e92-80b0-1a9e628c4bc3. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T22:33:38.361Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 9e33461ee1485c753c5625e257a26aa5523cb3391c80d28376b4c46e79bb378b. Source text hash: 6c5e98bf3163206a357e76efcf1ded12030b07e031d59bd304ccb5be750425a4. ## Revision and source-check history ### Version 1 — 2026-11-12T22:33:38.361Z Anthony Rhodes and Gadi Singer introduce the Behavioral Misalignment Metric Suite (BMMS), which evaluates whether a language model follows a particular policy corpus rather than scoring only preset behaviors such as hallucination or toxicity. Its measures include the Compliance Attainment Score (CAS), for severity-weighted attainment of binding obligations, and the Compliance Robustness Margin (CRM), for how much attainment survives adversarial pressure. The authors say a measure should be admitted only if its changes reflect the system being assessed, not the judging instrument. Across 1,003 obligations from eleven documents in five registers, the criterion reached 94.4% precision against blinded human judgment on vendor specifications, but 50.0% on analytical guidance. Re-scoring with judges from two other providers preserved target ordering, though one target’s absolute score varied. Comparing two detailed behavioral specifications, the authors report no resolved own-specification advantage after adjusting for document difficulty. They separately report that GPT-4o failed 70.7% of hard constraints in OpenAI’s Model Spec, while the two Claude models failed 0.0% and 9.1% of their respective constraints. The results distinguish relative advantage from absolute compliance, but the reported comparisons cover a limited set of documents and systems. ## Original source arXiv cs.AI · Article Original source date: Nov 6, 2026 (2026-11-06). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-12T22:33:38.361Z. First observed: 2026-11-09T05:04:28.621Z. Retrieved: 2026-11-12T22:32:44.451Z. - [BMMS: Policy-Conditioned Measurement of Behavioral Compliance in Language Models](https://arxiv.org/abs/2611.09147) ## Limits The abstract reports tests across eleven documents and comparisons involving the two most detailed published behavioral specifications; it does not establish how the results generalize beyond those materials. The reported precision differs by register: 94.4% on vendor specifications and 50.0% on analytical guidance, so the criterion’s performance is not uniform across these categories. Re-scoring preserved target ordering, but the absolute score for a single target varied by up to 0.22 across judges from other providers. The abstract reports model-specific constraint failure rates; these figures alone do not establish performance in real-world use. ## Open question How would BMMS’s admission criterion and compliance comparisons perform on policy documents and systems beyond the two detailed specifications evaluated? ## Source-stated Rhodes and Singer introduce BMMS as a way to measure a language model’s compliance with a supplied policy corpus. > We introduce the Behavioral Misalignment Metric Suite (BMMS), which measures policy-conditioned misalignment by taking a policy corpus as an argument - [Supporting source](https://arxiv.org/abs/2611.09147) ## Source-stated The authors define an admission test intended to distinguish changes in the system being assessed from changes caused by the judging instrument. > a quantity that moves when the judge changes, holding the target fixed, is measuring the instrument. - [Supporting source](https://arxiv.org/abs/2611.09147) ## Source-stated Across 1,003 obligations from eleven documents in five registers, the criterion reached 94.4% precision on vendor specifications and 50.0% on analytical guidance, against blinded human judgment. > Across 1,003 obligations from eleven documents in five registers, the admission criterion holds 94.4% precision against blinded human judgment on vendor specifications and 50.0% on analytical guidance - [Supporting source](https://arxiv.org/abs/2611.09147) ## Source-stated The authors report that own-specification advantage was not resolved from zero after adjusting for document difficulty, while absolute self-compliance differed across the systems assessed. > Own-specification advantage is not resolved from zero once document difficulty is subtracted, but absolute self-compliance is another matter - [Supporting source](https://arxiv.org/abs/2611.09147) ## Source-stated In the reported Model Spec comparison, GPT-4o failed 70.7% of its binding hard constraints; the two Claude models failed 0.0% and 9.1% of theirs. > GPT-4o fails 70.7% of the hard constraints binding it in the Model Spec published by OpenAI, where the two Claude models fail 0.0% and 9.1% of theirs. - [Supporting source](https://arxiv.org/abs/2611.09147) ## Source and contributor context Rhodes, Anthony, Singer, Gadi · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## AI alignment Research and practice aimed at making AI systems pursue the goals their builders and users intend. - [Definition reference](https://en.wikipedia.org/wiki/AI_alignment) ## Hallucination When an AI system states something false or made up as if it were true. - [Definition reference](https://developers.google.com/machine-learning/glossary#hallucination) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## OpenAI OpenAI develops AI models and products and publishes research, release announcements and evaluations. - [Explore this context](/entities/openai) - [Reference](https://openai.com/news/) ## Claude Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding. - [Explore this context](/entities/claude) - [Reference](https://claude.com/product/overview) - [Reference](https://www.anthropic.com/news) - [Reference](https://www.anthropic.com/research/team/economics) ## Editorial provenance Version 1. Publication ID: 2d3a140e-f4ad-4e92-80b0-1a9e628c4bc3. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T22:33:38.361Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 9e33461ee1485c753c5625e257a26aa5523cb3391c80d28376b4c46e79bb378b. Source text hash: 6c5e98bf3163206a357e76efcf1ded12030b07e031d59bd304ccb5be750425a4. ### uncertain — 2026-11-12T22:33:38.361Z — anthropic/claude-opus-5.5 Publication: 2d3a140e-f4ad-4e92-80b0-1a9e628c4bc3 attribution — pass: The draft correctly attributes BMMS, its metrics, and all reported results to authors Anthony Rhodes and Gadi Singer. It frames the findings as the authors' reports, not as established fact. > Authors:Anthony Rhodes, Gadi Singer summary — uncertain: The summary is faithful but hard for a non-specialist to follow. 'Own-specification advantage' is central and never explained; a minimal repair is to say it means whether a model scores better on its own developer's spec. 'Five registers' and 'precision against blinded human judgment' are also unexplained. The draft also leaves out a pointed detail: applying the admission test excluded one of the suite's own reported quantities. One short added clause would restore that. > Applying it excluded one of the quantities the suite itself reported. date — pass: The originalPublished date of 2026-11-06 matches the v1 submission date on arXiv. The later feed date is correctly not used. > \[Submitted on 6 Nov 2026\] support — pass: The claims, numbers and limitations match the abstract: 1,003 obligations, 94.4% versus 50.0% precision, the 0.22 score spread, and the 70.7%, 0.0% and 9.1% failure rates. The limitations and the question stay within the evidence and do not turn the test scores into real-world reliability. > re-scoring under judges from two other providers leaves target ordering invariant while the absolute score of a single target spans up to 0.22 media — pass: The selected media is the verified arXiv abstract page, which is the direct original source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/78745fe13b82676c8b68f74b4299c9b528430168dc6c81f5afdb536075f80577/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-paper-introduces-bmms-for-measuring-policy-78745fe1.json)