Bae, Jinhyung · Nov 18, 2026
arXiv cs.AI: Grading Executable Legal Rules Against Independent Records
Markdown · JSON · Original source
In an paper, Jinhyung Bae argues that legal programs can appear more accurate when judged against an answer written by someone who read the same law, rather than against records of what happened. The paper introduces LexTemporis, which grades programs for Korean SME status against company disclosures and the versions of rules in force at the relevant times. Bae reports that a transitional provision outside the main statutory text raises agreement with those disclosures from 83–86% to 89–90%. The disclosures also favor Bae’s reading of a contested provision in 130 of 189 firm-years where interpretations differ. Four wrote programs using different selections of law; a pre-registered comparison found no condition best across all four. In exploratory analyses, the apparent benefit of supplying the right provisions was smaller when programs were graded against records rather than Bae’s reading, for three models. The paper cautions that grading against a grader’s own reading can overstate how well legal programs work. Its evidence centers on Korean SME disclosures; the abstract also says an appendix adds comparison cases from US income tax and German hospital minimum volumes.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 18, 2026 (2026-11-18).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-19T06:02:06.984Z. First observed: 2026-11-19T05:04:09.191Z. Retrieved: 2026-11-19T06:01:13.468Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The supplied evidence is the page and abstract, not the paper's full methods or results; the reported comparisons cannot be assessed in greater detail here. The abstract says comparison cases from US income tax and German hospital minimum volumes are in an appendix but does not report their findings.
Open question
What findings does the appendix report from the US income tax and German hospital minimum-volume comparison cases?
Source-stated
The paper argues that grading legal programs against an answer based on the same law can reward agreement with the grader’s interpretation rather than correctness against what happened.
a program can score well by sharing the grader's reading rather than by getting the law right
Source-stated
LexTemporis grades programs for Korean SME status against company disclosures and the versions of the governing rules.
10,901 firm-years of SME status disclosed by Korean listed companies, with every version of the governing rules
Source-stated
The paper reports that a transitional provision raises agreement with disclosures from 83–86% to 89–90%.
one transitional provision raises agreement with the disclosures from 83-86% to 89-90%
Source-stated
Disclosures favor the author's reading of a contested provision in 130 of 189 firm-years where readings diverge.
favouring it in 130 of 189 firm-years where readings diverge
Source-stated
A pre-registered comparison found no condition better across all four LLMs that wrote the legal rules.
A pre-registered comparison finds no condition better across all four models.
Source-stated
In exploratory analyses, the apparent advantage of supplying the right provisions shrinks for three models when grading uses records instead of the author's reading.
the advantage shrinks for three models
Source-stated
The paper concludes that grading legal programs against a grader's own reading can overstate their performance.
Grading legal programs against the grader's own reading can overstate how well they work.
Source and contributor context
Bae, Jinhyung · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: e46c1277-8d1c-4ed3-88b2-6f65fb592c52.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-19T06:02:06.984Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 88d0cb66fc77b07909576aa8a22869761528bbbc525f1e06051bd550eb887d45. Source text hash: 345e1dea2f4936967bfdef16e3fe3f3f8f759776082d76ae2b7df55fde86295f.
Revision and source-check history
Version 1 · 2026-11-19T06:02:06.984Z
arXiv cs.AI: Grading Executable Legal Rules Against Independent Records
In an paper, Jinhyung Bae argues that legal programs can appear more accurate when judged against an answer written by someone who read the same law, rather than against records of what happened. The paper introduces LexTemporis, which grades programs for Korean SME status against company disclosures and the versions of rules in force at the relevant times. Bae reports that a transitional provision outside the main statutory text raises agreement with those disclosures from 83–86% to 89–90%. The disclosures also favor Bae’s reading of a contested provision in 130 of 189 firm-years where interpretations differ. Four wrote programs using different selections of law; a pre-registered comparison found no condition best across all four. In exploratory analyses, the apparent benefit of supplying the right provisions was smaller when programs were graded against records rather than Bae’s reading, for three models. The paper cautions that grading against a grader’s own reading can overstate how well legal programs work. Its evidence centers on Korean SME disclosures; the abstract also says an appendix adds comparison cases from US income tax and German hospital minimum volumes.
Original source
cs.AI · Article
Source details
Original source date: Nov 18, 2026 (2026-11-18).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-19T06:02:06.984Z. First observed: 2026-11-19T05:04:09.191Z. Retrieved: 2026-11-19T06:01:13.468Z.
Limits
The supplied evidence is the page and abstract, not the paper's full methods or results; the reported comparisons cannot be assessed in greater detail here. The abstract says comparison cases from US income tax and German hospital minimum volumes are in an appendix but does not report their findings.
Open question
What findings does the appendix report from the US income tax and German hospital minimum-volume comparison cases?
Source-stated
The paper argues that grading legal programs against an answer based on the same law can reward agreement with the grader’s interpretation rather than correctness against what happened.
a program can score well by sharing the grader's reading rather than by getting the law right
Source-stated
LexTemporis grades programs for Korean SME status against company disclosures and the versions of the governing rules.
10,901 firm-years of SME status disclosed by Korean listed companies, with every version of the governing rules
Source-stated
The paper reports that a transitional provision raises agreement with disclosures from 83–86% to 89–90%.
one transitional provision raises agreement with the disclosures from 83-86% to 89-90%
Source-stated
Disclosures favor the author's reading of a contested provision in 130 of 189 firm-years where readings diverge.
favouring it in 130 of 189 firm-years where readings diverge
Source-stated
A pre-registered comparison found no condition better across all four LLMs that wrote the legal rules.
A pre-registered comparison finds no condition better across all four models.
Source-stated
In exploratory analyses, the apparent advantage of supplying the right provisions shrinks for three models when grading uses records instead of the author's reading.
the advantage shrinks for three models
Source-stated
The paper concludes that grading legal programs against a grader's own reading can overstate their performance.
Grading legal programs against the grader's own reading can overstate how well they work.
Source and contributor context
Bae, Jinhyung · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: e46c1277-8d1c-4ed3-88b2-6f65fb592c52.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-19T06:02:06.984Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 88d0cb66fc77b07909576aa8a22869761528bbbc525f1e06051bd550eb887d45. Source text hash: 345e1dea2f4936967bfdef16e3fe3f3f8f759776082d76ae2b7df55fde86295f.
passed · 2026-11-19T06:02:06.984Z · anthropic/claude-opus-5.5
Publication: e46c1277-8d1c-4ed3-88b2-6f65fb592c52
attribution — pass: The reading attributes the paper and its arguments to Jinhyung Bae on arXiv, consistent with the author line. Findings are framed as what the paper reports, argues or cautions. LexTemporis is correctly described as the paper's grading approach. The cs.AI label matches the publisher field, and the page shows the paper cross-listed under cs.AI.
Authors:Jinhyung Bae
summary — pass: The reading leads with the author's thesis: legal programs graded against the grader's own reading can look better than they are. It then explains the record-based alternative and keeps the key figures, the null pre-registered result and the exploratory status. The scope caveat is preserved. Minor gaps: SME, LLM and firm-years are not expanded inline, and the 'partly by construction' nuance is omitted. Neither prevents understanding, assuming term tooltips cover SME and LLM. The question asks about the appendix findings, a genuine unknown.
Grading legal programs against the grader's own reading can overstate how well they work.
date — pass: originalPublished 2026-11-18 matches the v1 submission date shown on the arXiv page. It is not the later feed date of 2026-11-19.
[Submitted on 18 Nov 2026]
support — pass: Each claim matches the abstract: the grader-reading problem, the 10,901 firm-years, the 83–86% to 89–90% change, the 130 of 189 split, the null pre-registered result, and the shrinking advantage for three models. Describing the rule versions as 'in force at the relevant times' is a fair gloss of 'every version of the governing rules'. No unsupported benefits, costs or forecasts are added, and the limitations accurately note that the appendix findings are absent from the abstract.
In exploratory analyses, supplying the right provisions looks better graded against the author's reading than against the records: the advantage shrinks for three models
media — pass: The selected media ID matches the only verified candidate, which is the direct arXiv abstract page for the source. That is the most direct and relevant access point.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv