# arXiv cs.AI: Grading Executable Legal Rules Against Independent Records

[Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-grading-executable-legal-rules-against-independent-e3b6ff16)

In an arXiv paper, Jinhyung Bae argues that legal programs can appear more accurate when judged against an answer written by someone who read the same law, rather than against records of what happened. The paper introduces LexTemporis, which grades programs for Korean SME status against company disclosures and the versions of rules in force at the relevant times. Bae reports that a transitional provision outside the main statutory text raises agreement with those disclosures from 83–86% to 89–90%. The disclosures also favor Bae’s reading of a contested provision in 130 of 189 firm-years where interpretations differ. Four LLMs wrote programs using different selections of law; a pre-registered comparison found no condition best across all four. In exploratory analyses, the apparent benefit of supplying the right provisions was smaller when programs were graded against records rather than Bae’s reading, for three models. The paper cautions that grading against a grader’s own reading can overstate how well legal programs work. Its evidence centers on Korean SME disclosures; the abstract also says an appendix adds comparison cases from US income tax and German hospital minimum volumes.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv cs.AI · Article

Original source date: Nov 18, 2026 (2026-11-18).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-19T06:02:06.984Z. First observed: 2026-11-19T05:04:09.191Z. Retrieved: 2026-11-19T06:01:13.468Z.

- [Grading Executable Legal Rules Against Independent Records: Transitional Provisions and Korean SME Status](https://arxiv.org/abs/2611.24055)

## Limits

The supplied evidence is the arXiv page and abstract, not the paper's full methods or results; the reported comparisons cannot be assessed in greater detail here. The abstract says comparison cases from US income tax and German hospital minimum volumes are in an appendix but does not report their findings.


## Open question

What findings does the appendix report from the US income tax and German hospital minimum-volume comparison cases?


## Source-stated

The paper argues that grading legal programs against an answer based on the same law can reward agreement with the grader’s interpretation rather than correctness against what happened.

> a program can score well by sharing the grader's reading rather than by getting the law right

- [Supporting source](https://arxiv.org/abs/2611.24055)

## Source-stated

LexTemporis grades programs for Korean SME status against company disclosures and the versions of the governing rules.

> 10,901 firm-years of SME status disclosed by Korean listed companies, with every version of the governing rules

- [Supporting source](https://arxiv.org/abs/2611.24055)

## Source-stated

The paper reports that a transitional provision raises agreement with disclosures from 83–86% to 89–90%.

> one transitional provision raises agreement with the disclosures from 83-86% to 89-90%

- [Supporting source](https://arxiv.org/abs/2611.24055)

## Source-stated

Disclosures favor the author's reading of a contested provision in 130 of 189 firm-years where readings diverge.

> favouring it in 130 of 189 firm-years where readings diverge

- [Supporting source](https://arxiv.org/abs/2611.24055)

## Source-stated

A pre-registered comparison found no condition better across all four LLMs that wrote the legal rules.

> A pre-registered comparison finds no condition better across all four models.

- [Supporting source](https://arxiv.org/abs/2611.24055)

## Source-stated

In exploratory analyses, the apparent advantage of supplying the right provisions shrinks for three models when grading uses records instead of the author's reading.

> the advantage shrinks for three models

- [Supporting source](https://arxiv.org/abs/2611.24055)

## Source-stated

The paper concludes that grading legal programs against a grader's own reading can overstate their performance.

> Grading legal programs against the grader's own reading can overstate how well they work.

- [Supporting source](https://arxiv.org/abs/2611.24055)

## Source and contributor context

Bae, Jinhyung · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: e46c1277-8d1c-4ed3-88b2-6f65fb592c52.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-19T06:02:06.984Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 88d0cb66fc77b07909576aa8a22869761528bbbc525f1e06051bd550eb887d45. Source text hash: 345e1dea2f4936967bfdef16e3fe3f3f8f759776082d76ae2b7df55fde86295f.


## Revision and source-check history

### Version 1 — 2026-11-19T06:02:06.984Z

In an arXiv paper, Jinhyung Bae argues that legal programs can appear more accurate when judged against an answer written by someone who read the same law, rather than against records of what happened. The paper introduces LexTemporis, which grades programs for Korean SME status against company disclosures and the versions of rules in force at the relevant times. Bae reports that a transitional provision outside the main statutory text raises agreement with those disclosures from 83–86% to 89–90%. The disclosures also favor Bae’s reading of a contested provision in 130 of 189 firm-years where interpretations differ. Four LLMs wrote programs using different selections of law; a pre-registered comparison found no condition best across all four. In exploratory analyses, the apparent benefit of supplying the right provisions was smaller when programs were graded against records rather than Bae’s reading, for three models. The paper cautions that grading against a grader’s own reading can overstate how well legal programs work. Its evidence centers on Korean SME disclosures; the abstract also says an appendix adds comparison cases from US income tax and German hospital minimum volumes.

## Original source

arXiv cs.AI · Article

Original source date: Nov 18, 2026 (2026-11-18).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-19T06:02:06.984Z. First observed: 2026-11-19T05:04:09.191Z. Retrieved: 2026-11-19T06:01:13.468Z.

- [Grading Executable Legal Rules Against Independent Records: Transitional Provisions and Korean SME Status](https://arxiv.org/abs/2611.24055)

## Limits

The supplied evidence is the arXiv page and abstract, not the paper's full methods or results; the reported comparisons cannot be assessed in greater detail here. The abstract says comparison cases from US income tax and German hospital minimum volumes are in an appendix but does not report their findings.


## Open question

What findings does the appendix report from the US income tax and German hospital minimum-volume comparison cases?


## Source-stated

The paper argues that grading legal programs against an answer based on the same law can reward agreement with the grader’s interpretation rather than correctness against what happened.

> a program can score well by sharing the grader's reading rather than by getting the law right

- [Supporting source](https://arxiv.org/abs/2611.24055)

## Source-stated

LexTemporis grades programs for Korean SME status against company disclosures and the versions of the governing rules.

> 10,901 firm-years of SME status disclosed by Korean listed companies, with every version of the governing rules

- [Supporting source](https://arxiv.org/abs/2611.24055)

## Source-stated

The paper reports that a transitional provision raises agreement with disclosures from 83–86% to 89–90%.

> one transitional provision raises agreement with the disclosures from 83-86% to 89-90%

- [Supporting source](https://arxiv.org/abs/2611.24055)

## Source-stated

Disclosures favor the author's reading of a contested provision in 130 of 189 firm-years where readings diverge.

> favouring it in 130 of 189 firm-years where readings diverge

- [Supporting source](https://arxiv.org/abs/2611.24055)

## Source-stated

A pre-registered comparison found no condition better across all four LLMs that wrote the legal rules.

> A pre-registered comparison finds no condition better across all four models.

- [Supporting source](https://arxiv.org/abs/2611.24055)

## Source-stated

In exploratory analyses, the apparent advantage of supplying the right provisions shrinks for three models when grading uses records instead of the author's reading.

> the advantage shrinks for three models

- [Supporting source](https://arxiv.org/abs/2611.24055)

## Source-stated

The paper concludes that grading legal programs against a grader's own reading can overstate their performance.

> Grading legal programs against the grader's own reading can overstate how well they work.

- [Supporting source](https://arxiv.org/abs/2611.24055)

## Source and contributor context

Bae, Jinhyung · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: e46c1277-8d1c-4ed3-88b2-6f65fb592c52.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-19T06:02:06.984Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 88d0cb66fc77b07909576aa8a22869761528bbbc525f1e06051bd550eb887d45. Source text hash: 345e1dea2f4936967bfdef16e3fe3f3f8f759776082d76ae2b7df55fde86295f.


### passed — 2026-11-19T06:02:06.984Z — anthropic/claude-opus-5.5

Publication: e46c1277-8d1c-4ed3-88b2-6f65fb592c52

attribution — pass: The reading attributes the paper and its arguments to Jinhyung Bae on arXiv, consistent with the author line. Findings are framed as what the paper reports, argues or cautions. LexTemporis is correctly described as the paper's grading approach. The cs.AI label matches the publisher field, and the page shows the paper cross-listed under cs.AI.

> Authors:Jinhyung Bae

summary — pass: The reading leads with the author's thesis: legal programs graded against the grader's own reading can look better than they are. It then explains the record-based alternative and keeps the key figures, the null pre-registered result and the exploratory status. The scope caveat is preserved. Minor gaps: SME, LLM and firm-years are not expanded inline, and the 'partly by construction' nuance is omitted. Neither prevents understanding, assuming term tooltips cover SME and LLM. The question asks about the appendix findings, a genuine unknown.

> Grading legal programs against the grader's own reading can overstate how well they work.

date — pass: originalPublished 2026-11-18 matches the v1 submission date shown on the arXiv page. It is not the later feed date of 2026-11-19.

> \[Submitted on 18 Nov 2026\]

support — pass: Each claim matches the abstract: the grader-reading problem, the 10,901 firm-years, the 83–86% to 89–90% change, the 130 of 189 split, the null pre-registered result, and the shrinking advantage for three models. Describing the rule versions as 'in force at the relevant times' is a fair gloss of 'every version of the governing rules'. No unsupported benefits, costs or forecasts are added, and the limitations accurately note that the appendix findings are absent from the abstract.

> In exploratory analyses, supplying the right provisions looks better graded against the author's reading than against the records: the advantage shrinks for three models

media — pass: The selected media ID matches the only verified candidate, which is the direct arXiv abstract page for the source. That is the most direct and relevant access point.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/e3b6ff165011e132e5667903878e59681a12ccca80931a5ba7b210b86039244c/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-grading-executable-legal-rules-against-independent-e3b6ff16.json)
