Kouteili, Sam, Liang, William, Murray, Keeran, Zhou, Adriana, Viswanathan, Ramesh, Shapiro, Scott, Piskac, Ruzica · Oct 28, 2026
Deogen: Automatic Benchmark Generation for Reasoning over Policies
Markdown · JSON · Original source
The authors of Deogen present DeoGen, a system for automatically generating that test whether language models can apply policy rules to a situation and reach a verdict. They call this ability deontic reasoning. Their method turns policy rules into first-order logic and uses decision trees to generate cases: some have enough information for a verdict, while others test whether a model abstains when information is insufficient. The authors report that return the correct verdict 89-92% of the on their evaluations, but get only 22-40% correct on entries without enough information to claim a verdict, even when instructed to abstain. The contrast matters because applying rules includes recognizing when the facts do not support a decision. DeoGen is intended to make benchmark creation less manual and to generate many cases from each tree. The reported results concern these benchmarks; the abstract does not establish how performance transfers to real regulated settings. How do the authors’ benchmark results compare with models’ performance on real policy decisions?
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 28, 2026 (2026-10-28).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-13T07:18:21.116Z. First observed: 2026-10-30T06:45:28.309Z. Retrieved: 2026-10-31T11:03:18.690Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The supplied evidence is the paper’s abstract; it does not give evaluation details, model names, or the full results. The abstract does not establish whether the reported benchmark performance transfers to real decisions in regulated settings. The abstract does not describe how policy rules are selected or how their formalization is checked.
Open question
How do the authors’ results compare with models’ performance on real policy decisions?
Source-stated
The authors present DeoGen as an automatic, scalable, parametrizable way to generate benchmarks for deontic reasoning.
In this paper, we present DeoGen, a scalable, automatic, and parametrizable benchmark generator for deontic reasoning.
Source-stated
DeoGen turns policy rules into first-order logic and uses decision trees to create benchmark entries, including cases where the model should abstain because information is insufficient.
By autoformalizing policy rules to first-order logic, we can construct decision trees that capture possible outcome trajectories.
Source-stated
The authors report frontier models return the correct verdict 89-92% of the time in their evaluations.
Evaluations show frontier models return the correct verdict 89-92% of the time.
Source-stated
The authors report much lower correctness on cases with insufficient information, even when models are explicitly instructed to abstain.
performance degrades (22-40% correct) on entries without sufficient information to claim a verdict, even with explicit instructions to abstain in such cases.
Source and contributor context
Kouteili, Sam, Liang, William, Murray, Keeran, Zhou, Adriana, Viswanathan, Ramesh, Shapiro, Scott, Piskac, Ruzica · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 931944d6-ecec-4b75-8c34-6c20f9d9c55d.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-13T07:18:21.116Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: b9a8e22b90e8bb76be649f09e49418d03756a8e8663364629f1e81ee49a8527b. Source text hash: 2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1.
Revision and source-check history
Version 1 · 2026-11-13T07:18:21.116Z
Deogen: Automatic Benchmark Generation for Reasoning over Policies
The authors of Deogen present DeoGen, a system for automatically generating that test whether language models can apply policy rules to a situation and reach a verdict. They call this ability deontic reasoning. Their method turns policy rules into first-order logic and uses decision trees to generate cases: some have enough information for a verdict, while others test whether a model abstains when information is insufficient. The authors report that return the correct verdict 89-92% of the on their evaluations, but get only 22-40% correct on entries without enough information to claim a verdict, even when instructed to abstain. The contrast matters because applying rules includes recognizing when the facts do not support a decision. DeoGen is intended to make benchmark creation less manual and to generate many cases from each tree. The reported results concern these benchmarks; the abstract does not establish how performance transfers to real regulated settings. How do the authors’ benchmark results compare with models’ performance on real policy decisions?
Original source
cs.AI · Article
Source details
Original source date: Oct 28, 2026 (2026-10-28).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-13T07:18:21.116Z. First observed: 2026-10-30T06:45:28.309Z. Retrieved: 2026-10-31T11:03:18.690Z.
Limits
The supplied evidence is the paper’s abstract; it does not give evaluation details, model names, or the full results. The abstract does not establish whether the reported benchmark performance transfers to real decisions in regulated settings. The abstract does not describe how policy rules are selected or how their formalization is checked.
Open question
How do the authors’ results compare with models’ performance on real policy decisions?
Source-stated
The authors present DeoGen as an automatic, scalable, parametrizable way to generate benchmarks for deontic reasoning.
In this paper, we present DeoGen, a scalable, automatic, and parametrizable benchmark generator for deontic reasoning.
Source-stated
DeoGen turns policy rules into first-order logic and uses decision trees to create benchmark entries, including cases where the model should abstain because information is insufficient.
By autoformalizing policy rules to first-order logic, we can construct decision trees that capture possible outcome trajectories.
Source-stated
The authors report frontier models return the correct verdict 89-92% of the time in their evaluations.
Evaluations show frontier models return the correct verdict 89-92% of the time.
Source-stated
The authors report much lower correctness on cases with insufficient information, even when models are explicitly instructed to abstain.
performance degrades (22-40% correct) on entries without sufficient information to claim a verdict, even with explicit instructions to abstain in such cases.
Source and contributor context
Kouteili, Sam, Liang, William, Murray, Keeran, Zhou, Adriana, Viswanathan, Ramesh, Shapiro, Scott, Piskac, Ruzica · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 931944d6-ecec-4b75-8c34-6c20f9d9c55d.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-13T07:18:21.116Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: b9a8e22b90e8bb76be649f09e49418d03756a8e8663364629f1e81ee49a8527b. Source text hash: 2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1.
passed · 2026-11-13T07:18:21.116Z · anthropic/claude-opus-5.5
Publication: 931944d6-ecec-4b75-8c34-6c20f9d9c55d
attribution — pass: The reading attributes DeoGen and its results to the paper's authors. Performance figures are framed as their reported evaluations rather than established fact.
Authors:Sam Kouteili, William Liang, Keeran Murray, Adriana Zhou, Ramesh Viswanathan, Scott Shapiro, Ruzica Piskac
summary — pass: The summary leads with what the authors built. It defines deontic reasoning in plain terms, explains the method, and gives the abstract's central contrast between verdict accuracy and abstention accuracy. 'First-order logic' and 'frontier models' are not glossed and would suit term-library tooltips, but the core meaning stays readable. The question follows from the noted gap about real-world transfer.
where a model must apply given rules to the facts of a situation to determine a verdict
date — pass: originalPublished 2026-10-28 matches the arXiv v1 submission date. The later feed entry date is correctly not used.
[Submitted on 28 Oct 2026]
support — pass: Each claim matches the abstract, including the 89-92% and 22-40% figures and the explicit-instruction condition. The point that applying rules includes recognizing insufficient facts follows from the abstract's framing of non-leaf paths. The reading avoids claiming real-world reliability and explicitly notes the transfer gap.
Paths ending at a leaf entail a verdict, while non-leaf paths measure a model's ability to abstain when presented with insufficient information.
media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the supplied candidate and the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference