{"id":"cf121beba68e16e5b6acde7c1b50abfe30956bb5b46b36613694c9755705fe06","title":"Deogen: Automatic Benchmark Generation for Reasoning over Policies","argument":"The authors of Deogen present DeoGen, a system for automatically generating benchmarks that test whether language models can apply policy rules to a situation and reach a verdict. They call this ability deontic reasoning. Their method turns policy rules into first-order logic and uses decision trees to generate cases: some have enough information for a verdict, while others test whether a model abstains when information is insufficient.\n\nThe authors report that frontier models return the correct verdict 89-92% of the time on their evaluations, but get only 22-40% correct on entries without enough information to claim a verdict, even when instructed to abstain. The contrast matters because applying rules includes recognizing when the facts do not support a decision. DeoGen is intended to make benchmark creation less manual and to generate many cases from each tree. The reported results concern these benchmarks; the abstract does not establish how performance transfers to real regulated settings. How do the authors’ benchmark results compare with models’ performance on real policy decisions?","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The supplied evidence is the paper’s abstract; it does not give evaluation details, model names, or the full benchmark results. The abstract does not establish whether the reported benchmark performance transfers to real decisions in regulated settings. The abstract does not describe how policy rules are selected or how their formalization is checked.","speaker":{"name":"Kouteili, Sam, Liang, William, Murray, Keeran, Zhou, Adriana, Viswanathan, Ramesh, Shapiro, Scott, Piskac, Ruzica","roleAtSource":"Article author"},"source":{"title":"Deogen: Automatic Benchmark Generation for Reasoning over Policies","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.38687","publishedAt":"2026-10-28","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors present DeoGen as an automatic, scalable, parametrizable way to generate benchmarks for deontic reasoning.","excerpt":"In this paper, we present DeoGen, a scalable, automatic, and parametrizable benchmark generator for deontic reasoning.","url":"https://arxiv.org/abs/2610.38687","relationship":"Source-stated"},{"text":"DeoGen turns policy rules into first-order logic and uses decision trees to create benchmark entries, including cases where the model should abstain because information is insufficient.","excerpt":"By autoformalizing policy rules to first-order logic, we can construct decision trees that capture possible outcome trajectories.","url":"https://arxiv.org/abs/2610.38687","relationship":"Source-stated"},{"text":"The authors report frontier models return the correct verdict 89-92% of the time in their evaluations.","excerpt":"Evaluations show frontier models return the correct verdict 89-92% of the time.","url":"https://arxiv.org/abs/2610.38687","relationship":"Source-stated"},{"text":"The authors report much lower correctness on cases with insufficient information, even when models are explicitly instructed to abstain.","excerpt":"performance degrades (22-40% correct) on entries without sufficient information to claim a verdict, even with explicit instructions to abstain in such cases.","url":"https://arxiv.org/abs/2610.38687","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.38687","checkedAt":"2026-11-13T09:26:07.541Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How do the authors’ benchmark results compare with models’ performance on real policy decisions?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Deogen: Automatic Benchmark Generation for Reasoning over Policies","summary":"The authors of Deogen present DeoGen, a system for automatically generating benchmarks that test whether language models can apply policy rules to a situation and reach a verdict. They call this ability deontic reasoning. Their method turns policy rules into first-order logic and uses decision trees to generate cases: some have enough information for a verdict, while others test whether a model abstains when information is insufficient.\n\nThe authors report that frontier models return the correct verdict 89-92% of the time on their evaluations, but get only 22-40% correct on entries without enough information to claim a verdict, even when instructed to abstain. The contrast matters because applying rules includes recognizing when the facts do not support a decision. DeoGen is intended to make benchmark creation less manual and to generate many cases from each tree. The reported results concern these benchmarks; the abstract does not establish how performance transfers to real regulated settings. How do the authors’ benchmark results compare with models’ performance on real policy decisions?","question":"How do the authors’ benchmark results compare with models’ performance on real policy decisions?","claims":[{"text":"The authors present DeoGen as an automatic, scalable, parametrizable way to generate benchmarks for deontic reasoning.","excerpt":"In this paper, we present DeoGen, a scalable, automatic, and parametrizable benchmark generator for deontic reasoning.","support":"source-stated"},{"text":"DeoGen turns policy rules into first-order logic and uses decision trees to create benchmark entries, including cases where the model should abstain because information is insufficient.","excerpt":"By autoformalizing policy rules to first-order logic, we can construct decision trees that capture possible outcome trajectories.","support":"source-stated"},{"text":"The authors report frontier models return the correct verdict 89-92% of the time in their evaluations.","excerpt":"Evaluations show frontier models return the correct verdict 89-92% of the time.","support":"source-stated"},{"text":"The authors report much lower correctness on cases with insufficient information, even when models are explicitly instructed to abstain.","excerpt":"performance degrades (22-40% correct) on entries without sufficient information to claim a verdict, even with explicit instructions to abstain in such cases.","support":"source-stated"}],"limitations":["The supplied evidence is the paper’s abstract; it does not give evaluation details, model names, or the full benchmark results.","The abstract does not establish whether the reported benchmark performance transfers to real decisions in regulated settings.","The abstract does not describe how policy rules are selected or how their formalization is checked."],"mediaId":"2845e815975a4f7bda7625622b4f3200465bf230ee94011d635c894fdc2325ba","correctionReason":null,"id":"cf121beba68e16e5b6acde7c1b50abfe30956bb5b46b36613694c9755705fe06","publicationId":"931944d6-ecec-4b75-8c34-6c20f9d9c55d","version":1,"sourceUrl":"https://arxiv.org/abs/2610.38687","sourceTitle":"Deogen: Automatic Benchmark Generation for Reasoning over Policies","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-28","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-30T06:45:28.309Z","createdAt":"2026-11-13T07:18:21.116Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"2845e815975a4f7bda7625622b4f3200465bf230ee94011d635c894fdc2325ba","kind":"article","url":"https://arxiv.org/abs/2610.38687","sourceUrl":"https://arxiv.org/abs/2610.38687","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-31T11:03:18.913Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.38687","checkedAt":"2026-11-13T09:26:07.541Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"2845e815975a4f7bda7625622b4f3200465bf230ee94011d635c894fdc2325ba","confidence":0.8200000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"cee09ae28c7fe5278eee374d84f112d86ac9d15b6abf2d60b638e63117eb5418","decidedAt":"2026-10-31T11:03:20.480Z"},"provenance":{"textHash":"2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1","observationId":"b9a8e22b90e8bb76be649f09e49418d03756a8e8663364629f1e81ee49a8527b","retrievedAt":"2026-10-31T11:03:18.690Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"a63867c1-9693-4bd7-88c2-51a93aa2dcca"},"reviewer":{"model":"gpt-6-luna","runId":"896a34ad-197c-4ff7-8436-fe2970bce10b"}},"audit":{"status":"passed","reviewedAt":"2026-11-13T07:18:21.116Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Kouteili, Sam","url":null,"kind":"unknown"},{"name":"Liang, William","url":null,"kind":"unknown"},{"name":"Murray, Keeran","url":null,"kind":"unknown"},{"name":"Zhou, Adriana","url":null,"kind":"unknown"},{"name":"Viswanathan, Ramesh","url":null,"kind":"unknown"},{"name":"Shapiro, Scott","url":null,"kind":"unknown"},{"name":"Piskac, Ruzica","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Kouteili, Sam","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Kouteili, Sam","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38687","locator":"head/meta[name=author|citation_author]"},"assertionId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T09:26:07.541Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97","sourceUrl":"https://arxiv.org/abs/2610.38687","contentSha256":"5241e633c7707880b9e483c0c6e63e1aa945fc3cabaa325fdc6afc57fe80a9d9","textHash":"2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1","observedAt":"2026-11-13T09:26:07.541Z"}},{"actor":{"name":"Liang, William","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liang, William","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38687","locator":"head/meta[name=author|citation_author]"},"assertionId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T09:26:07.541Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97","sourceUrl":"https://arxiv.org/abs/2610.38687","contentSha256":"5241e633c7707880b9e483c0c6e63e1aa945fc3cabaa325fdc6afc57fe80a9d9","textHash":"2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1","observedAt":"2026-11-13T09:26:07.541Z"}},{"actor":{"name":"Murray, Keeran","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Murray, Keeran","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38687","locator":"head/meta[name=author|citation_author]"},"assertionId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T09:26:07.541Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97","sourceUrl":"https://arxiv.org/abs/2610.38687","contentSha256":"5241e633c7707880b9e483c0c6e63e1aa945fc3cabaa325fdc6afc57fe80a9d9","textHash":"2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1","observedAt":"2026-11-13T09:26:07.541Z"}},{"actor":{"name":"Zhou, Adriana","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhou, Adriana","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38687","locator":"head/meta[name=author|citation_author]"},"assertionId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T09:26:07.541Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97","sourceUrl":"https://arxiv.org/abs/2610.38687","contentSha256":"5241e633c7707880b9e483c0c6e63e1aa945fc3cabaa325fdc6afc57fe80a9d9","textHash":"2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1","observedAt":"2026-11-13T09:26:07.541Z"}},{"actor":{"name":"Viswanathan, Ramesh","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Viswanathan, Ramesh","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38687","locator":"head/meta[name=author|citation_author]"},"assertionId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T09:26:07.541Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97","sourceUrl":"https://arxiv.org/abs/2610.38687","contentSha256":"5241e633c7707880b9e483c0c6e63e1aa945fc3cabaa325fdc6afc57fe80a9d9","textHash":"2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1","observedAt":"2026-11-13T09:26:07.541Z"}},{"actor":{"name":"Shapiro, Scott","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Shapiro, Scott","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38687","locator":"head/meta[name=author|citation_author]"},"assertionId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T09:26:07.541Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97","sourceUrl":"https://arxiv.org/abs/2610.38687","contentSha256":"5241e633c7707880b9e483c0c6e63e1aa945fc3cabaa325fdc6afc57fe80a9d9","textHash":"2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1","observedAt":"2026-11-13T09:26:07.541Z"}},{"actor":{"name":"Piskac, Ruzica","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Piskac, Ruzica","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38687","locator":"head/meta[name=author|citation_author]"},"assertionId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T09:26:07.541Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97","sourceUrl":"https://arxiv.org/abs/2610.38687","contentSha256":"5241e633c7707880b9e483c0c6e63e1aa945fc3cabaa325fdc6afc57fe80a9d9","textHash":"2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1","observedAt":"2026-11-13T09:26:07.541Z"}}],"participationCoverage":{"total":7,"returned":7,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.38687","snapshotId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97","contentSha256":"5241e633c7707880b9e483c0c6e63e1aa945fc3cabaa325fdc6afc57fe80a9d9","textHash":"2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1","observedAt":"2026-11-13T09:26:07.541Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/cf121beba68e16e5b6acde7c1b50abfe30956bb5b46b36613694c9755705fe06/source"}},"canonicalUrl":"https://escapemeridian.com/readings/deogen-automatic-benchmark-generation-for-reasoning-over-cf121beb","formats":{"html":"/readings/deogen-automatic-benchmark-generation-for-reasoning-over-cf121beb","markdown":"/readings/deogen-automatic-benchmark-generation-for-reasoning-over-cf121beb.md","json":"/readings/deogen-automatic-benchmark-generation-for-reasoning-over-cf121beb.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-frontier-model","name":"Frontier model","explanation":"One of the most capable AI models available at a given time; the set shifts as models improve.","referenceUrl":"https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"},{"id":"time","name":"TIME","kind":"organization","context":"TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.","referenceUrls":["https://time.com/"],"path":"/entities/time","json":"/entities/time.json","markdown":"/entities/time.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"cf121beba68e16e5b6acde7c1b50abfe30956bb5b46b36613694c9755705fe06","revision":"931944d6-ecec-4b75-8c34-6c20f9d9c55d","revisionKind":"publication-id","title":"Deogen: Automatic Benchmark Generation for Reasoning over Policies","url":"https://escapemeridian.com/readings/deogen-automatic-benchmark-generation-for-reasoning-over-cf121beb"},"source":{"type":"source","id":"https://arxiv.org/abs/2610.38687","revision":"2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1","revisionKind":"sha256-source-text","title":"Deogen: Automatic Benchmark Generation for Reasoning over Policies","url":"https://arxiv.org/abs/2610.38687"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"cf121beba68e16e5b6acde7c1b50abfe30956bb5b46b36613694c9755705fe06","title":"Deogen: Automatic Benchmark Generation for Reasoning over Policies","argument":"The authors of Deogen present DeoGen, a system for automatically generating benchmarks that test whether language models can apply policy rules to a situation and reach a verdict. They call this ability deontic reasoning. Their method turns policy rules into first-order logic and uses decision trees to generate cases: some have enough information for a verdict, while others test whether a model abstains when information is insufficient.\n\nThe authors report that frontier models return the correct verdict 89-92% of the time on their evaluations, but get only 22-40% correct on entries without enough information to claim a verdict, even when instructed to abstain. The contrast matters because applying rules includes recognizing when the facts do not support a decision. DeoGen is intended to make benchmark creation less manual and to generate many cases from each tree. The reported results concern these benchmarks; the abstract does not establish how performance transfers to real regulated settings. How do the authors’ benchmark results compare with models’ performance on real policy decisions?","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The supplied evidence is the paper’s abstract; it does not give evaluation details, model names, or the full benchmark results. The abstract does not establish whether the reported benchmark performance transfers to real decisions in regulated settings. The abstract does not describe how policy rules are selected or how their formalization is checked.","speaker":{"name":"Kouteili, Sam, Liang, William, Murray, Keeran, Zhou, Adriana, Viswanathan, Ramesh, Shapiro, Scott, Piskac, Ruzica","roleAtSource":"Article author"},"source":{"title":"Deogen: Automatic Benchmark Generation for Reasoning over Policies","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.38687","publishedAt":"2026-10-28","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors present DeoGen as an automatic, scalable, parametrizable way to generate benchmarks for deontic reasoning.","excerpt":"In this paper, we present DeoGen, a scalable, automatic, and parametrizable benchmark generator for deontic reasoning.","url":"https://arxiv.org/abs/2610.38687","relationship":"Source-stated"},{"text":"DeoGen turns policy rules into first-order logic and uses decision trees to create benchmark entries, including cases where the model should abstain because information is insufficient.","excerpt":"By autoformalizing policy rules to first-order logic, we can construct decision trees that capture possible outcome trajectories.","url":"https://arxiv.org/abs/2610.38687","relationship":"Source-stated"},{"text":"The authors report frontier models return the correct verdict 89-92% of the time in their evaluations.","excerpt":"Evaluations show frontier models return the correct verdict 89-92% of the time.","url":"https://arxiv.org/abs/2610.38687","relationship":"Source-stated"},{"text":"The authors report much lower correctness on cases with insufficient information, even when models are explicitly instructed to abstain.","excerpt":"performance degrades (22-40% correct) on entries without sufficient information to claim a verdict, even with explicit instructions to abstain in such cases.","url":"https://arxiv.org/abs/2610.38687","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.38687","checkedAt":"2026-11-13T09:26:07.541Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How do the authors’ benchmark results compare with models’ performance on real policy decisions?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Deogen: Automatic Benchmark Generation for Reasoning over Policies","summary":"The authors of Deogen present DeoGen, a system for automatically generating benchmarks that test whether language models can apply policy rules to a situation and reach a verdict. They call this ability deontic reasoning. Their method turns policy rules into first-order logic and uses decision trees to generate cases: some have enough information for a verdict, while others test whether a model abstains when information is insufficient.\n\nThe authors report that frontier models return the correct verdict 89-92% of the time on their evaluations, but get only 22-40% correct on entries without enough information to claim a verdict, even when instructed to abstain. The contrast matters because applying rules includes recognizing when the facts do not support a decision. DeoGen is intended to make benchmark creation less manual and to generate many cases from each tree. The reported results concern these benchmarks; the abstract does not establish how performance transfers to real regulated settings. How do the authors’ benchmark results compare with models’ performance on real policy decisions?","question":"How do the authors’ benchmark results compare with models’ performance on real policy decisions?","claims":[{"text":"The authors present DeoGen as an automatic, scalable, parametrizable way to generate benchmarks for deontic reasoning.","excerpt":"In this paper, we present DeoGen, a scalable, automatic, and parametrizable benchmark generator for deontic reasoning.","support":"source-stated"},{"text":"DeoGen turns policy rules into first-order logic and uses decision trees to create benchmark entries, including cases where the model should abstain because information is insufficient.","excerpt":"By autoformalizing policy rules to first-order logic, we can construct decision trees that capture possible outcome trajectories.","support":"source-stated"},{"text":"The authors report frontier models return the correct verdict 89-92% of the time in their evaluations.","excerpt":"Evaluations show frontier models return the correct verdict 89-92% of the time.","support":"source-stated"},{"text":"The authors report much lower correctness on cases with insufficient information, even when models are explicitly instructed to abstain.","excerpt":"performance degrades (22-40% correct) on entries without sufficient information to claim a verdict, even with explicit instructions to abstain in such cases.","support":"source-stated"}],"limitations":["The supplied evidence is the paper’s abstract; it does not give evaluation details, model names, or the full benchmark results.","The abstract does not establish whether the reported benchmark performance transfers to real decisions in regulated settings.","The abstract does not describe how policy rules are selected or how their formalization is checked."],"mediaId":"2845e815975a4f7bda7625622b4f3200465bf230ee94011d635c894fdc2325ba","correctionReason":null,"id":"cf121beba68e16e5b6acde7c1b50abfe30956bb5b46b36613694c9755705fe06","publicationId":"931944d6-ecec-4b75-8c34-6c20f9d9c55d","version":1,"sourceUrl":"https://arxiv.org/abs/2610.38687","sourceTitle":"Deogen: Automatic Benchmark Generation for Reasoning over Policies","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-28","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-30T06:45:28.309Z","createdAt":"2026-11-13T07:18:21.116Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"2845e815975a4f7bda7625622b4f3200465bf230ee94011d635c894fdc2325ba","kind":"article","url":"https://arxiv.org/abs/2610.38687","sourceUrl":"https://arxiv.org/abs/2610.38687","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-31T11:03:18.913Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.38687","checkedAt":"2026-11-13T09:26:07.541Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"2845e815975a4f7bda7625622b4f3200465bf230ee94011d635c894fdc2325ba","confidence":0.8200000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"cee09ae28c7fe5278eee374d84f112d86ac9d15b6abf2d60b638e63117eb5418","decidedAt":"2026-10-31T11:03:20.480Z"},"provenance":{"textHash":"2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1","observationId":"b9a8e22b90e8bb76be649f09e49418d03756a8e8663364629f1e81ee49a8527b","retrievedAt":"2026-10-31T11:03:18.690Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"a63867c1-9693-4bd7-88c2-51a93aa2dcca"},"reviewer":{"model":"gpt-6-luna","runId":"896a34ad-197c-4ff7-8436-fe2970bce10b"}},"audit":{"status":"passed","reviewedAt":"2026-11-13T07:18:21.116Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Kouteili, Sam","url":null,"kind":"unknown"},{"name":"Liang, William","url":null,"kind":"unknown"},{"name":"Murray, Keeran","url":null,"kind":"unknown"},{"name":"Zhou, Adriana","url":null,"kind":"unknown"},{"name":"Viswanathan, Ramesh","url":null,"kind":"unknown"},{"name":"Shapiro, Scott","url":null,"kind":"unknown"},{"name":"Piskac, Ruzica","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Kouteili, Sam","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Kouteili, Sam","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38687","locator":"head/meta[name=author|citation_author]"},"assertionId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T09:26:07.541Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97","sourceUrl":"https://arxiv.org/abs/2610.38687","contentSha256":"5241e633c7707880b9e483c0c6e63e1aa945fc3cabaa325fdc6afc57fe80a9d9","textHash":"2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1","observedAt":"2026-11-13T09:26:07.541Z"}},{"actor":{"name":"Liang, William","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liang, William","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38687","locator":"head/meta[name=author|citation_author]"},"assertionId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T09:26:07.541Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97","sourceUrl":"https://arxiv.org/abs/2610.38687","contentSha256":"5241e633c7707880b9e483c0c6e63e1aa945fc3cabaa325fdc6afc57fe80a9d9","textHash":"2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1","observedAt":"2026-11-13T09:26:07.541Z"}},{"actor":{"name":"Murray, Keeran","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Murray, Keeran","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38687","locator":"head/meta[name=author|citation_author]"},"assertionId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T09:26:07.541Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97","sourceUrl":"https://arxiv.org/abs/2610.38687","contentSha256":"5241e633c7707880b9e483c0c6e63e1aa945fc3cabaa325fdc6afc57fe80a9d9","textHash":"2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1","observedAt":"2026-11-13T09:26:07.541Z"}},{"actor":{"name":"Zhou, Adriana","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhou, Adriana","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38687","locator":"head/meta[name=author|citation_author]"},"assertionId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T09:26:07.541Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97","sourceUrl":"https://arxiv.org/abs/2610.38687","contentSha256":"5241e633c7707880b9e483c0c6e63e1aa945fc3cabaa325fdc6afc57fe80a9d9","textHash":"2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1","observedAt":"2026-11-13T09:26:07.541Z"}},{"actor":{"name":"Viswanathan, Ramesh","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Viswanathan, Ramesh","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38687","locator":"head/meta[name=author|citation_author]"},"assertionId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T09:26:07.541Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97","sourceUrl":"https://arxiv.org/abs/2610.38687","contentSha256":"5241e633c7707880b9e483c0c6e63e1aa945fc3cabaa325fdc6afc57fe80a9d9","textHash":"2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1","observedAt":"2026-11-13T09:26:07.541Z"}},{"actor":{"name":"Shapiro, Scott","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Shapiro, Scott","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38687","locator":"head/meta[name=author|citation_author]"},"assertionId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T09:26:07.541Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97","sourceUrl":"https://arxiv.org/abs/2610.38687","contentSha256":"5241e633c7707880b9e483c0c6e63e1aa945fc3cabaa325fdc6afc57fe80a9d9","textHash":"2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1","observedAt":"2026-11-13T09:26:07.541Z"}},{"actor":{"name":"Piskac, Ruzica","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Piskac, Ruzica","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.38687","locator":"head/meta[name=author|citation_author]"},"assertionId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T09:26:07.541Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97","sourceUrl":"https://arxiv.org/abs/2610.38687","contentSha256":"5241e633c7707880b9e483c0c6e63e1aa945fc3cabaa325fdc6afc57fe80a9d9","textHash":"2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1","observedAt":"2026-11-13T09:26:07.541Z"}}],"participationCoverage":{"total":7,"returned":7,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.38687","snapshotId":"1efcfdb1a4be08b029a0455bc6acf1e26cf8ce3c2caf8d42de8bb959b8cfac97","contentSha256":"5241e633c7707880b9e483c0c6e63e1aa945fc3cabaa325fdc6afc57fe80a9d9","textHash":"2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1","observedAt":"2026-11-13T09:26:07.541Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/cf121beba68e16e5b6acde7c1b50abfe30956bb5b46b36613694c9755705fe06/source"}}}],"audits":[{"publicationId":"931944d6-ecec-4b75-8c34-6c20f9d9c55d","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes DeoGen and its results to the paper's authors. Performance figures are framed as their reported evaluations rather than established fact.","excerpt":"Authors:Sam Kouteili, William Liang, Keeran Murray, Adriana Zhou, Ramesh Viswanathan, Scott Shapiro, Ruzica Piskac"},{"dimension":"summary","status":"pass","explanation":"The summary leads with what the authors built. It defines deontic reasoning in plain terms, explains the method, and gives the abstract's central contrast between verdict accuracy and abstention accuracy. 'First-order logic' and 'frontier models' are not glossed and would suit term-library tooltips, but the core meaning stays readable. The question follows from the noted gap about real-world transfer.","excerpt":"where a model must apply given rules to the facts of a situation to determine a verdict"},{"dimension":"date","status":"pass","explanation":"originalPublished 2026-10-28 matches the arXiv v1 submission date. The later feed entry date is correctly not used.","excerpt":"[Submitted on 28 Oct 2026]"},{"dimension":"support","status":"pass","explanation":"Each claim matches the abstract, including the 89-92% and 22-40% figures and the explicit-instruction condition. The point that applying rules includes recognizing insufficient facts follows from the abstract's framing of non-leaf paths. The reading avoids claiming real-world reliability and explicitly notes the transfer gap.","excerpt":"Paths ending at a leaf entail a verdict, while non-leaf paths measure a model's ability to abstain when presented with insufficient information."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct arXiv abstract page for this paper. It is the supplied candidate and the most direct access to the source.","excerpt":null}],"createdAt":"2026-11-13T07:18:21.116Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}