# Deogen: Automatic Benchmark Generation for Reasoning over Policies [Canonical reading](https://escapemeridian.com/readings/deogen-automatic-benchmark-generation-for-reasoning-over-cf121beb) The authors of Deogen present DeoGen, a system for automatically generating benchmarks that test whether language models can apply policy rules to a situation and reach a verdict. They call this ability deontic reasoning. Their method turns policy rules into first-order logic and uses decision trees to generate cases: some have enough information for a verdict, while others test whether a model abstains when information is insufficient. The authors report that frontier models return the correct verdict 89-92% of the time on their evaluations, but get only 22-40% correct on entries without enough information to claim a verdict, even when instructed to abstain. The contrast matters because applying rules includes recognizing when the facts do not support a decision. DeoGen is intended to make benchmark creation less manual and to generate many cases from each tree. The reported results concern these benchmarks; the abstract does not establish how performance transfers to real regulated settings. How do the authors’ benchmark results compare with models’ performance on real policy decisions? Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Oct 28, 2026 (2026-10-28). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-13T07:18:21.116Z. First observed: 2026-10-30T06:45:28.309Z. Retrieved: 2026-10-31T11:03:18.690Z. - [Deogen: Automatic Benchmark Generation for Reasoning over Policies](https://arxiv.org/abs/2610.38687) ## Limits The supplied evidence is the paper’s abstract; it does not give evaluation details, model names, or the full benchmark results. The abstract does not establish whether the reported benchmark performance transfers to real decisions in regulated settings. The abstract does not describe how policy rules are selected or how their formalization is checked. ## Open question How do the authors’ benchmark results compare with models’ performance on real policy decisions? ## Source-stated The authors present DeoGen as an automatic, scalable, parametrizable way to generate benchmarks for deontic reasoning. > In this paper, we present DeoGen, a scalable, automatic, and parametrizable benchmark generator for deontic reasoning. - [Supporting source](https://arxiv.org/abs/2610.38687) ## Source-stated DeoGen turns policy rules into first-order logic and uses decision trees to create benchmark entries, including cases where the model should abstain because information is insufficient. > By autoformalizing policy rules to first-order logic, we can construct decision trees that capture possible outcome trajectories. - [Supporting source](https://arxiv.org/abs/2610.38687) ## Source-stated The authors report frontier models return the correct verdict 89-92% of the time in their evaluations. > Evaluations show frontier models return the correct verdict 89-92% of the time. - [Supporting source](https://arxiv.org/abs/2610.38687) ## Source-stated The authors report much lower correctness on cases with insufficient information, even when models are explicitly instructed to abstain. > performance degrades (22-40% correct) on entries without sufficient information to claim a verdict, even with explicit instructions to abstain in such cases. - [Supporting source](https://arxiv.org/abs/2610.38687) ## Source and contributor context Kouteili, Sam, Liang, William, Murray, Keeran, Zhou, Adriana, Viswanathan, Ramesh, Shapiro, Scott, Piskac, Ruzica · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Frontier model One of the most capable AI models available at a given time; the set shifts as models improve. - [Definition reference](https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## TIME TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services. - [Explore this context](/entities/time) - [Reference](https://time.com/) ## Editorial provenance Version 1. Publication ID: 931944d6-ecec-4b75-8c34-6c20f9d9c55d. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-13T07:18:21.116Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: b9a8e22b90e8bb76be649f09e49418d03756a8e8663364629f1e81ee49a8527b. Source text hash: 2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1. ## Revision and source-check history ### Version 1 — 2026-11-13T07:18:21.116Z The authors of Deogen present DeoGen, a system for automatically generating benchmarks that test whether language models can apply policy rules to a situation and reach a verdict. They call this ability deontic reasoning. Their method turns policy rules into first-order logic and uses decision trees to generate cases: some have enough information for a verdict, while others test whether a model abstains when information is insufficient. The authors report that frontier models return the correct verdict 89-92% of the time on their evaluations, but get only 22-40% correct on entries without enough information to claim a verdict, even when instructed to abstain. The contrast matters because applying rules includes recognizing when the facts do not support a decision. DeoGen is intended to make benchmark creation less manual and to generate many cases from each tree. The reported results concern these benchmarks; the abstract does not establish how performance transfers to real regulated settings. How do the authors’ benchmark results compare with models’ performance on real policy decisions? ## Original source arXiv cs.AI · Article Original source date: Oct 28, 2026 (2026-10-28). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-13T07:18:21.116Z. First observed: 2026-10-30T06:45:28.309Z. Retrieved: 2026-10-31T11:03:18.690Z. - [Deogen: Automatic Benchmark Generation for Reasoning over Policies](https://arxiv.org/abs/2610.38687) ## Limits The supplied evidence is the paper’s abstract; it does not give evaluation details, model names, or the full benchmark results. The abstract does not establish whether the reported benchmark performance transfers to real decisions in regulated settings. The abstract does not describe how policy rules are selected or how their formalization is checked. ## Open question How do the authors’ benchmark results compare with models’ performance on real policy decisions? ## Source-stated The authors present DeoGen as an automatic, scalable, parametrizable way to generate benchmarks for deontic reasoning. > In this paper, we present DeoGen, a scalable, automatic, and parametrizable benchmark generator for deontic reasoning. - [Supporting source](https://arxiv.org/abs/2610.38687) ## Source-stated DeoGen turns policy rules into first-order logic and uses decision trees to create benchmark entries, including cases where the model should abstain because information is insufficient. > By autoformalizing policy rules to first-order logic, we can construct decision trees that capture possible outcome trajectories. - [Supporting source](https://arxiv.org/abs/2610.38687) ## Source-stated The authors report frontier models return the correct verdict 89-92% of the time in their evaluations. > Evaluations show frontier models return the correct verdict 89-92% of the time. - [Supporting source](https://arxiv.org/abs/2610.38687) ## Source-stated The authors report much lower correctness on cases with insufficient information, even when models are explicitly instructed to abstain. > performance degrades (22-40% correct) on entries without sufficient information to claim a verdict, even with explicit instructions to abstain in such cases. - [Supporting source](https://arxiv.org/abs/2610.38687) ## Source and contributor context Kouteili, Sam, Liang, William, Murray, Keeran, Zhou, Adriana, Viswanathan, Ramesh, Shapiro, Scott, Piskac, Ruzica · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Frontier model One of the most capable AI models available at a given time; the set shifts as models improve. - [Definition reference](https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## TIME TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services. - [Explore this context](/entities/time) - [Reference](https://time.com/) ## Editorial provenance Version 1. Publication ID: 931944d6-ecec-4b75-8c34-6c20f9d9c55d. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-13T07:18:21.116Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: b9a8e22b90e8bb76be649f09e49418d03756a8e8663364629f1e81ee49a8527b. Source text hash: 2eaf817b66f59e115984bfbba7ac7151f0e493fc06dec0317a89e53f857e86c1. ### passed — 2026-11-13T07:18:21.116Z — anthropic/claude-opus-5.5 Publication: 931944d6-ecec-4b75-8c34-6c20f9d9c55d attribution — pass: The reading attributes DeoGen and its results to the paper's authors. Performance figures are framed as their reported evaluations rather than established fact. > Authors:Sam Kouteili, William Liang, Keeran Murray, Adriana Zhou, Ramesh Viswanathan, Scott Shapiro, Ruzica Piskac summary — pass: The summary leads with what the authors built. It defines deontic reasoning in plain terms, explains the method, and gives the abstract's central contrast between verdict accuracy and abstention accuracy. 'First-order logic' and 'frontier models' are not glossed and would suit term-library tooltips, but the core meaning stays readable. The question follows from the noted gap about real-world transfer. > where a model must apply given rules to the facts of a situation to determine a verdict date — pass: originalPublished 2026-10-28 matches the arXiv v1 submission date. The later feed entry date is correctly not used. > \[Submitted on 28 Oct 2026\] support — pass: Each claim matches the abstract, including the 89-92% and 22-40% figures and the explicit-instruction condition. The point that applying rules includes recognizing insufficient facts follows from the abstract's framing of non-leaf paths. The reading avoids claiming real-world reliability and explicitly notes the transfer gap. > Paths ending at a leaf entail a verdict, while non-leaf paths measure a model's ability to abstain when presented with insufficient information. media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the supplied candidate and the most direct access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/cf121beba68e16e5b6acde7c1b50abfe30956bb5b46b36613694c9755705fe06/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/deogen-automatic-benchmark-generation-for-reasoning-over-cf121beb.json)