{"id":"78745fe13b82676c8b68f74b4299c9b528430168dc6c81f5afdb536075f80577","title":"arXiv cs.AI paper introduces BMMS for measuring policy compliance","argument":"Anthony Rhodes and Gadi Singer introduce the Behavioral Misalignment Metric Suite (BMMS), which evaluates whether a language model follows a particular policy corpus rather than scoring only preset behaviors such as hallucination or toxicity. Its measures include the Compliance Attainment Score (CAS), for severity-weighted attainment of binding obligations, and the Compliance Robustness Margin (CRM), for how much attainment survives adversarial pressure. The authors say a measure should be admitted only if its changes reflect the system being assessed, not the judging instrument. Across 1,003 obligations from eleven documents in five registers, the criterion reached 94.4% precision against blinded human judgment on vendor specifications, but 50.0% on analytical guidance. Re-scoring with judges from two other providers preserved target ordering, though one target’s absolute score varied. Comparing two detailed behavioral specifications, the authors report no resolved own-specification advantage after adjusting for document difficulty. They separately report that GPT-4o failed 70.7% of hard constraints in OpenAI’s Model Spec, while the two Claude models failed 0.0% and 9.1% of their respective constraints. The results distinguish relative advantage from absolute compliance, but the reported comparisons cover a limited set of documents and systems.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract reports tests across eleven documents and comparisons involving the two most detailed published behavioral specifications; it does not establish how the results generalize beyond those materials. The reported precision differs by register: 94.4% on vendor specifications and 50.0% on analytical guidance, so the criterion’s performance is not uniform across these categories. Re-scoring preserved target ordering, but the absolute score for a single target varied by up to 0.22 across judges from other providers. The abstract reports model-specific constraint failure rates; these figures alone do not establish performance in real-world use.","speaker":{"name":"Rhodes, Anthony, Singer, Gadi","roleAtSource":"Article author"},"source":{"title":"BMMS: Policy-Conditioned Measurement of Behavioral Compliance in Language Models","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.09147","publishedAt":"2026-11-06","dateBasis":"Date basis not provided."},"evidence":[{"text":"Rhodes and Singer introduce BMMS as a way to measure a language model’s compliance with a supplied policy corpus.","excerpt":"We introduce the Behavioral Misalignment Metric Suite (BMMS), which measures policy-conditioned misalignment by taking a policy corpus as an argument","url":"https://arxiv.org/abs/2611.09147","relationship":"Source-stated"},{"text":"The authors define an admission test intended to distinguish changes in the system being assessed from changes caused by the judging instrument.","excerpt":"a quantity that moves when the judge changes, holding the target fixed, is measuring the instrument.","url":"https://arxiv.org/abs/2611.09147","relationship":"Source-stated"},{"text":"Across 1,003 obligations from eleven documents in five registers, the criterion reached 94.4% precision on vendor specifications and 50.0% on analytical guidance, against blinded human judgment.","excerpt":"Across 1,003 obligations from eleven documents in five registers, the admission criterion holds 94.4% precision against blinded human judgment on vendor specifications and 50.0% on analytical guidance","url":"https://arxiv.org/abs/2611.09147","relationship":"Source-stated"},{"text":"The authors report that own-specification advantage was not resolved from zero after adjusting for document difficulty, while absolute self-compliance differed across the systems assessed.","excerpt":"Own-specification advantage is not resolved from zero once document difficulty is subtracted, but absolute self-compliance is another matter","url":"https://arxiv.org/abs/2611.09147","relationship":"Source-stated"},{"text":"In the reported Model Spec comparison, GPT-4o failed 70.7% of its binding hard constraints; the two Claude models failed 0.0% and 9.1% of theirs.","excerpt":"GPT-4o fails 70.7% of the hard constraints binding it in the Model Spec published by OpenAI, where the two Claude models fail 0.0% and 9.1% of theirs.","url":"https://arxiv.org/abs/2611.09147","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.09147","checkedAt":"2026-11-13T15:36:10.628Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How would BMMS’s admission criterion and compliance comparisons perform on policy documents and systems beyond the two detailed specifications evaluated?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI paper introduces BMMS for measuring policy compliance","summary":"Anthony Rhodes and Gadi Singer introduce the Behavioral Misalignment Metric Suite (BMMS), which evaluates whether a language model follows a particular policy corpus rather than scoring only preset behaviors such as hallucination or toxicity. Its measures include the Compliance Attainment Score (CAS), for severity-weighted attainment of binding obligations, and the Compliance Robustness Margin (CRM), for how much attainment survives adversarial pressure. The authors say a measure should be admitted only if its changes reflect the system being assessed, not the judging instrument. Across 1,003 obligations from eleven documents in five registers, the criterion reached 94.4% precision against blinded human judgment on vendor specifications, but 50.0% on analytical guidance. Re-scoring with judges from two other providers preserved target ordering, though one target’s absolute score varied. Comparing two detailed behavioral specifications, the authors report no resolved own-specification advantage after adjusting for document difficulty. They separately report that GPT-4o failed 70.7% of hard constraints in OpenAI’s Model Spec, while the two Claude models failed 0.0% and 9.1% of their respective constraints. The results distinguish relative advantage from absolute compliance, but the reported comparisons cover a limited set of documents and systems.","question":"How would BMMS’s admission criterion and compliance comparisons perform on policy documents and systems beyond the two detailed specifications evaluated?","claims":[{"text":"Rhodes and Singer introduce BMMS as a way to measure a language model’s compliance with a supplied policy corpus.","excerpt":"We introduce the Behavioral Misalignment Metric Suite (BMMS), which measures policy-conditioned misalignment by taking a policy corpus as an argument","support":"source-stated"},{"text":"The authors define an admission test intended to distinguish changes in the system being assessed from changes caused by the judging instrument.","excerpt":"a quantity that moves when the judge changes, holding the target fixed, is measuring the instrument.","support":"source-stated"},{"text":"Across 1,003 obligations from eleven documents in five registers, the criterion reached 94.4% precision on vendor specifications and 50.0% on analytical guidance, against blinded human judgment.","excerpt":"Across 1,003 obligations from eleven documents in five registers, the admission criterion holds 94.4% precision against blinded human judgment on vendor specifications and 50.0% on analytical guidance","support":"source-stated"},{"text":"The authors report that own-specification advantage was not resolved from zero after adjusting for document difficulty, while absolute self-compliance differed across the systems assessed.","excerpt":"Own-specification advantage is not resolved from zero once document difficulty is subtracted, but absolute self-compliance is another matter","support":"source-stated"},{"text":"In the reported Model Spec comparison, GPT-4o failed 70.7% of its binding hard constraints; the two Claude models failed 0.0% and 9.1% of theirs.","excerpt":"GPT-4o fails 70.7% of the hard constraints binding it in the Model Spec published by OpenAI, where the two Claude models fail 0.0% and 9.1% of theirs.","support":"source-stated"}],"limitations":["The abstract reports tests across eleven documents and comparisons involving the two most detailed published behavioral specifications; it does not establish how the results generalize beyond those materials.","The reported precision differs by register: 94.4% on vendor specifications and 50.0% on analytical guidance, so the criterion’s performance is not uniform across these categories.","Re-scoring preserved target ordering, but the absolute score for a single target varied by up to 0.22 across judges from other providers.","The abstract reports model-specific constraint failure rates; these figures alone do not establish performance in real-world use."],"mediaId":"c23967f483c16b958b896218ab8b1e2ad572e9031cfa2fad88186e98e93dcf61","correctionReason":null,"id":"78745fe13b82676c8b68f74b4299c9b528430168dc6c81f5afdb536075f80577","publicationId":"2d3a140e-f4ad-4e92-80b0-1a9e628c4bc3","version":1,"sourceUrl":"https://arxiv.org/abs/2611.09147","sourceTitle":"BMMS: Policy-Conditioned Measurement of Behavioral Compliance in Language Models","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-06","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-09T05:04:28.621Z","createdAt":"2026-11-12T22:33:38.361Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"c23967f483c16b958b896218ab8b1e2ad572e9031cfa2fad88186e98e93dcf61","kind":"article","url":"https://arxiv.org/abs/2611.09147","sourceUrl":"https://arxiv.org/abs/2611.09147","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T22:32:44.537Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.09147","checkedAt":"2026-11-13T15:36:10.628Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"c23967f483c16b958b896218ab8b1e2ad572e9031cfa2fad88186e98e93dcf61","confidence":0.72,"provider":"typesafe","model":"jev-1.13.0","inputHash":"15ae0b5a242a429def7c1b4a9ad10e396fe3189e28ff0aab6b223b29c7b65ae3","decidedAt":"2026-11-12T22:32:45.624Z"},"provenance":{"textHash":"6c5e98bf3163206a357e76efcf1ded12030b07e031d59bd304ccb5be750425a4","observationId":"9e33461ee1485c753c5625e257a26aa5523cb3391c80d28376b4c46e79bb378b","retrievedAt":"2026-11-12T22:32:44.451Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"9c193b55-d6bf-4386-85ff-66264261173c"},"reviewer":{"model":"gpt-6-luna","runId":"22985cbd-04c4-42dd-83b0-6a9e1ceda7ea"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-12T22:33:38.361Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Rhodes, Anthony","url":null,"kind":"unknown"},{"name":"Singer, Gadi","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Rhodes, Anthony","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Rhodes, Anthony","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09147","locator":"head/meta[name=author|citation_author]"},"assertionId":"1c48550b0ae00012e1a923c3e31ec284e284d7562538223f4011f64213a9b1a0:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T15:36:10.628Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1c48550b0ae00012e1a923c3e31ec284e284d7562538223f4011f64213a9b1a0","sourceUrl":"https://arxiv.org/abs/2611.09147","contentSha256":"b8c91def416fc25448b8462504c978ce5f4a103bcfbeffbcc7d225d19d4450d2","textHash":"6c5e98bf3163206a357e76efcf1ded12030b07e031d59bd304ccb5be750425a4","observedAt":"2026-11-13T15:36:10.628Z"}},{"actor":{"name":"Singer, Gadi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Singer, Gadi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09147","locator":"head/meta[name=author|citation_author]"},"assertionId":"1c48550b0ae00012e1a923c3e31ec284e284d7562538223f4011f64213a9b1a0:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T15:36:10.628Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1c48550b0ae00012e1a923c3e31ec284e284d7562538223f4011f64213a9b1a0","sourceUrl":"https://arxiv.org/abs/2611.09147","contentSha256":"b8c91def416fc25448b8462504c978ce5f4a103bcfbeffbcc7d225d19d4450d2","textHash":"6c5e98bf3163206a357e76efcf1ded12030b07e031d59bd304ccb5be750425a4","observedAt":"2026-11-13T15:36:10.628Z"}}],"participationCoverage":{"total":2,"returned":2,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.09147","snapshotId":"1c48550b0ae00012e1a923c3e31ec284e284d7562538223f4011f64213a9b1a0","contentSha256":"b8c91def416fc25448b8462504c978ce5f4a103bcfbeffbcc7d225d19d4450d2","textHash":"6c5e98bf3163206a357e76efcf1ded12030b07e031d59bd304ccb5be750425a4","observedAt":"2026-11-13T15:36:10.628Z","basis":"publisher-page-metadata","sourceLinks":41,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/78745fe13b82676c8b68f74b4299c9b528430168dc6c81f5afdb536075f80577/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-paper-introduces-bmms-for-measuring-policy-78745fe1","formats":{"html":"/readings/arxiv-cs-ai-paper-introduces-bmms-for-measuring-policy-78745fe1","markdown":"/readings/arxiv-cs-ai-paper-introduces-bmms-for-measuring-policy-78745fe1.md","json":"/readings/arxiv-cs-ai-paper-introduces-bmms-for-measuring-policy-78745fe1.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-ai-alignment","name":"AI alignment","explanation":"Research and practice aimed at making AI systems pursue the goals their builders and users intend.","referenceUrl":"https://en.wikipedia.org/wiki/AI_alignment"},{"id":"term-hallucination","name":"Hallucination","explanation":"When an AI system states something false or made up as if it were true.","referenceUrl":"https://developers.google.com/machine-learning/glossary#hallucination"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"},{"id":"openai","name":"OpenAI","kind":"organization","context":"OpenAI develops AI models and products and publishes research, release announcements and evaluations.","referenceUrls":["https://openai.com/news/"],"path":"/entities/openai","json":"/entities/openai.json","markdown":"/entities/openai.md"},{"id":"claude","name":"Claude","kind":"model","context":"Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.","referenceUrls":["https://claude.com/product/overview","https://www.anthropic.com/news","https://www.anthropic.com/research/team/economics"],"path":"/entities/claude","json":"/entities/claude.json","markdown":"/entities/claude.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"78745fe13b82676c8b68f74b4299c9b528430168dc6c81f5afdb536075f80577","revision":"2d3a140e-f4ad-4e92-80b0-1a9e628c4bc3","revisionKind":"publication-id","title":"arXiv cs.AI paper introduces BMMS for measuring policy compliance","url":"https://escapemeridian.com/readings/arxiv-cs-ai-paper-introduces-bmms-for-measuring-policy-78745fe1"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.09147","revision":"6c5e98bf3163206a357e76efcf1ded12030b07e031d59bd304ccb5be750425a4","revisionKind":"sha256-source-text","title":"BMMS: Policy-Conditioned Measurement of Behavioral Compliance in Language Models","url":"https://arxiv.org/abs/2611.09147"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"78745fe13b82676c8b68f74b4299c9b528430168dc6c81f5afdb536075f80577","title":"arXiv cs.AI paper introduces BMMS for measuring policy compliance","argument":"Anthony Rhodes and Gadi Singer introduce the Behavioral Misalignment Metric Suite (BMMS), which evaluates whether a language model follows a particular policy corpus rather than scoring only preset behaviors such as hallucination or toxicity. Its measures include the Compliance Attainment Score (CAS), for severity-weighted attainment of binding obligations, and the Compliance Robustness Margin (CRM), for how much attainment survives adversarial pressure. The authors say a measure should be admitted only if its changes reflect the system being assessed, not the judging instrument. Across 1,003 obligations from eleven documents in five registers, the criterion reached 94.4% precision against blinded human judgment on vendor specifications, but 50.0% on analytical guidance. Re-scoring with judges from two other providers preserved target ordering, though one target’s absolute score varied. Comparing two detailed behavioral specifications, the authors report no resolved own-specification advantage after adjusting for document difficulty. They separately report that GPT-4o failed 70.7% of hard constraints in OpenAI’s Model Spec, while the two Claude models failed 0.0% and 9.1% of their respective constraints. The results distinguish relative advantage from absolute compliance, but the reported comparisons cover a limited set of documents and systems.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract reports tests across eleven documents and comparisons involving the two most detailed published behavioral specifications; it does not establish how the results generalize beyond those materials. The reported precision differs by register: 94.4% on vendor specifications and 50.0% on analytical guidance, so the criterion’s performance is not uniform across these categories. Re-scoring preserved target ordering, but the absolute score for a single target varied by up to 0.22 across judges from other providers. The abstract reports model-specific constraint failure rates; these figures alone do not establish performance in real-world use.","speaker":{"name":"Rhodes, Anthony, Singer, Gadi","roleAtSource":"Article author"},"source":{"title":"BMMS: Policy-Conditioned Measurement of Behavioral Compliance in Language Models","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.09147","publishedAt":"2026-11-06","dateBasis":"Date basis not provided."},"evidence":[{"text":"Rhodes and Singer introduce BMMS as a way to measure a language model’s compliance with a supplied policy corpus.","excerpt":"We introduce the Behavioral Misalignment Metric Suite (BMMS), which measures policy-conditioned misalignment by taking a policy corpus as an argument","url":"https://arxiv.org/abs/2611.09147","relationship":"Source-stated"},{"text":"The authors define an admission test intended to distinguish changes in the system being assessed from changes caused by the judging instrument.","excerpt":"a quantity that moves when the judge changes, holding the target fixed, is measuring the instrument.","url":"https://arxiv.org/abs/2611.09147","relationship":"Source-stated"},{"text":"Across 1,003 obligations from eleven documents in five registers, the criterion reached 94.4% precision on vendor specifications and 50.0% on analytical guidance, against blinded human judgment.","excerpt":"Across 1,003 obligations from eleven documents in five registers, the admission criterion holds 94.4% precision against blinded human judgment on vendor specifications and 50.0% on analytical guidance","url":"https://arxiv.org/abs/2611.09147","relationship":"Source-stated"},{"text":"The authors report that own-specification advantage was not resolved from zero after adjusting for document difficulty, while absolute self-compliance differed across the systems assessed.","excerpt":"Own-specification advantage is not resolved from zero once document difficulty is subtracted, but absolute self-compliance is another matter","url":"https://arxiv.org/abs/2611.09147","relationship":"Source-stated"},{"text":"In the reported Model Spec comparison, GPT-4o failed 70.7% of its binding hard constraints; the two Claude models failed 0.0% and 9.1% of theirs.","excerpt":"GPT-4o fails 70.7% of the hard constraints binding it in the Model Spec published by OpenAI, where the two Claude models fail 0.0% and 9.1% of theirs.","url":"https://arxiv.org/abs/2611.09147","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.09147","checkedAt":"2026-11-13T15:36:10.628Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How would BMMS’s admission criterion and compliance comparisons perform on policy documents and systems beyond the two detailed specifications evaluated?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI paper introduces BMMS for measuring policy compliance","summary":"Anthony Rhodes and Gadi Singer introduce the Behavioral Misalignment Metric Suite (BMMS), which evaluates whether a language model follows a particular policy corpus rather than scoring only preset behaviors such as hallucination or toxicity. Its measures include the Compliance Attainment Score (CAS), for severity-weighted attainment of binding obligations, and the Compliance Robustness Margin (CRM), for how much attainment survives adversarial pressure. The authors say a measure should be admitted only if its changes reflect the system being assessed, not the judging instrument. Across 1,003 obligations from eleven documents in five registers, the criterion reached 94.4% precision against blinded human judgment on vendor specifications, but 50.0% on analytical guidance. Re-scoring with judges from two other providers preserved target ordering, though one target’s absolute score varied. Comparing two detailed behavioral specifications, the authors report no resolved own-specification advantage after adjusting for document difficulty. They separately report that GPT-4o failed 70.7% of hard constraints in OpenAI’s Model Spec, while the two Claude models failed 0.0% and 9.1% of their respective constraints. The results distinguish relative advantage from absolute compliance, but the reported comparisons cover a limited set of documents and systems.","question":"How would BMMS’s admission criterion and compliance comparisons perform on policy documents and systems beyond the two detailed specifications evaluated?","claims":[{"text":"Rhodes and Singer introduce BMMS as a way to measure a language model’s compliance with a supplied policy corpus.","excerpt":"We introduce the Behavioral Misalignment Metric Suite (BMMS), which measures policy-conditioned misalignment by taking a policy corpus as an argument","support":"source-stated"},{"text":"The authors define an admission test intended to distinguish changes in the system being assessed from changes caused by the judging instrument.","excerpt":"a quantity that moves when the judge changes, holding the target fixed, is measuring the instrument.","support":"source-stated"},{"text":"Across 1,003 obligations from eleven documents in five registers, the criterion reached 94.4% precision on vendor specifications and 50.0% on analytical guidance, against blinded human judgment.","excerpt":"Across 1,003 obligations from eleven documents in five registers, the admission criterion holds 94.4% precision against blinded human judgment on vendor specifications and 50.0% on analytical guidance","support":"source-stated"},{"text":"The authors report that own-specification advantage was not resolved from zero after adjusting for document difficulty, while absolute self-compliance differed across the systems assessed.","excerpt":"Own-specification advantage is not resolved from zero once document difficulty is subtracted, but absolute self-compliance is another matter","support":"source-stated"},{"text":"In the reported Model Spec comparison, GPT-4o failed 70.7% of its binding hard constraints; the two Claude models failed 0.0% and 9.1% of theirs.","excerpt":"GPT-4o fails 70.7% of the hard constraints binding it in the Model Spec published by OpenAI, where the two Claude models fail 0.0% and 9.1% of theirs.","support":"source-stated"}],"limitations":["The abstract reports tests across eleven documents and comparisons involving the two most detailed published behavioral specifications; it does not establish how the results generalize beyond those materials.","The reported precision differs by register: 94.4% on vendor specifications and 50.0% on analytical guidance, so the criterion’s performance is not uniform across these categories.","Re-scoring preserved target ordering, but the absolute score for a single target varied by up to 0.22 across judges from other providers.","The abstract reports model-specific constraint failure rates; these figures alone do not establish performance in real-world use."],"mediaId":"c23967f483c16b958b896218ab8b1e2ad572e9031cfa2fad88186e98e93dcf61","correctionReason":null,"id":"78745fe13b82676c8b68f74b4299c9b528430168dc6c81f5afdb536075f80577","publicationId":"2d3a140e-f4ad-4e92-80b0-1a9e628c4bc3","version":1,"sourceUrl":"https://arxiv.org/abs/2611.09147","sourceTitle":"BMMS: Policy-Conditioned Measurement of Behavioral Compliance in Language Models","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-06","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-09T05:04:28.621Z","createdAt":"2026-11-12T22:33:38.361Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"c23967f483c16b958b896218ab8b1e2ad572e9031cfa2fad88186e98e93dcf61","kind":"article","url":"https://arxiv.org/abs/2611.09147","sourceUrl":"https://arxiv.org/abs/2611.09147","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T22:32:44.537Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.09147","checkedAt":"2026-11-13T15:36:10.628Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"c23967f483c16b958b896218ab8b1e2ad572e9031cfa2fad88186e98e93dcf61","confidence":0.72,"provider":"typesafe","model":"jev-1.13.0","inputHash":"15ae0b5a242a429def7c1b4a9ad10e396fe3189e28ff0aab6b223b29c7b65ae3","decidedAt":"2026-11-12T22:32:45.624Z"},"provenance":{"textHash":"6c5e98bf3163206a357e76efcf1ded12030b07e031d59bd304ccb5be750425a4","observationId":"9e33461ee1485c753c5625e257a26aa5523cb3391c80d28376b4c46e79bb378b","retrievedAt":"2026-11-12T22:32:44.451Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"9c193b55-d6bf-4386-85ff-66264261173c"},"reviewer":{"model":"gpt-6-luna","runId":"22985cbd-04c4-42dd-83b0-6a9e1ceda7ea"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-12T22:33:38.361Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Rhodes, Anthony","url":null,"kind":"unknown"},{"name":"Singer, Gadi","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Rhodes, Anthony","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Rhodes, Anthony","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09147","locator":"head/meta[name=author|citation_author]"},"assertionId":"1c48550b0ae00012e1a923c3e31ec284e284d7562538223f4011f64213a9b1a0:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T15:36:10.628Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1c48550b0ae00012e1a923c3e31ec284e284d7562538223f4011f64213a9b1a0","sourceUrl":"https://arxiv.org/abs/2611.09147","contentSha256":"b8c91def416fc25448b8462504c978ce5f4a103bcfbeffbcc7d225d19d4450d2","textHash":"6c5e98bf3163206a357e76efcf1ded12030b07e031d59bd304ccb5be750425a4","observedAt":"2026-11-13T15:36:10.628Z"}},{"actor":{"name":"Singer, Gadi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Singer, Gadi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09147","locator":"head/meta[name=author|citation_author]"},"assertionId":"1c48550b0ae00012e1a923c3e31ec284e284d7562538223f4011f64213a9b1a0:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T15:36:10.628Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1c48550b0ae00012e1a923c3e31ec284e284d7562538223f4011f64213a9b1a0","sourceUrl":"https://arxiv.org/abs/2611.09147","contentSha256":"b8c91def416fc25448b8462504c978ce5f4a103bcfbeffbcc7d225d19d4450d2","textHash":"6c5e98bf3163206a357e76efcf1ded12030b07e031d59bd304ccb5be750425a4","observedAt":"2026-11-13T15:36:10.628Z"}}],"participationCoverage":{"total":2,"returned":2,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.09147","snapshotId":"1c48550b0ae00012e1a923c3e31ec284e284d7562538223f4011f64213a9b1a0","contentSha256":"b8c91def416fc25448b8462504c978ce5f4a103bcfbeffbcc7d225d19d4450d2","textHash":"6c5e98bf3163206a357e76efcf1ded12030b07e031d59bd304ccb5be750425a4","observedAt":"2026-11-13T15:36:10.628Z","basis":"publisher-page-metadata","sourceLinks":41,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/78745fe13b82676c8b68f74b4299c9b528430168dc6c81f5afdb536075f80577/source"}}}],"audits":[{"publicationId":"2d3a140e-f4ad-4e92-80b0-1a9e628c4bc3","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The draft correctly attributes BMMS, its metrics, and all reported results to authors Anthony Rhodes and Gadi Singer. It frames the findings as the authors' reports, not as established fact.","excerpt":"Authors:Anthony Rhodes, Gadi Singer"},{"dimension":"summary","status":"uncertain","explanation":"The summary is faithful but hard for a non-specialist to follow. 'Own-specification advantage' is central and never explained; a minimal repair is to say it means whether a model scores better on its own developer's spec. 'Five registers' and 'precision against blinded human judgment' are also unexplained. The draft also leaves out a pointed detail: applying the admission test excluded one of the suite's own reported quantities. One short added clause would restore that.","excerpt":"Applying it excluded one of the quantities the suite itself reported."},{"dimension":"date","status":"pass","explanation":"The originalPublished date of 2026-11-06 matches the v1 submission date on arXiv. The later feed date is correctly not used.","excerpt":"[Submitted on 6 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"The claims, numbers and limitations match the abstract: 1,003 obligations, 94.4% versus 50.0% precision, the 0.22 score spread, and the 70.7%, 0.0% and 9.1% failure rates. The limitations and the question stay within the evidence and do not turn the test scores into real-world reliability.","excerpt":"re-scoring under judges from two other providers leaves target ordering invariant while the absolute score of a single target spans up to 0.22"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified arXiv abstract page, which is the direct original source.","excerpt":null}],"createdAt":"2026-11-12T22:33:38.361Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}