Dong, Zhaohe · Sep 24, 2026
arXiv cs.AI paper examines what sets the limits of automated audit
Markdown · JSON · Original source
Zhaohe Dong’s paper examines a shipped arrangement in which an auditor from one vendor checks another model’s work, alongside deterministic checks. On general-purpose code with hidden tests, eight readings reached 30.0% recall while flagging 16.0% of correct code; more readings brought diminishing returns, with no plateau. The result also shifted with the reading model and its sampling, a rulebook sentence, and the decision rule. A post hoc rating found that much of what was missed was behaviour the specification never determined. In three preregistered science studies, checking whether reported numbers traced to their sources flagged all 49 inconsistent reports and none of 25 consistent fabrications; the model auditor flagged 23 fabrications. For data, a documentation-derived validator and the auditor caught overlapping but different faulty items, together flagging 112 of 140, with a 2.9% flag rate on clean data. For scientific code, framing the task around one step reduced flags on correct code from 68.7% to 36.0%. Dong’s results make the limits and variability of automated audit concrete: outcomes depended on the task and its framing, and deterministic checks and model auditors caught different errors while each had limits.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Sep 24, 2026 (2026-09-24).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-19T08:46:55.383Z. First observed: 2026-11-19T05:04:13.242Z. Retrieved: 2026-11-19T08:34:22.422Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The evidence here is the paper’s abstract; it does not provide the detailed methods or results needed to assess the studies beyond the reported figures. The abstract says that two analyses were unregistered but does not identify them in the supplied text. The abstract’s post hoc rating describes much missed behaviour as unspecified by the specification; it does not establish how that rating was conducted.
Open question
How would audit results change if specifications determined behaviour that the post hoc rating found unspecified?
Source-stated
Dong studies a shipped arrangement combining a model auditor from a different vendor with deterministic checks.
an auditor from a different vendor reads the work and deterministic checks verify what code can verify
Source-stated
For general-purpose code with hidden tests, eight readings reached 30.0% recall and flagged 16.0% of correct code, with diminishing returns and no plateau.
eight readings of the auditor reach 30.0% recall [20.0, 40.7] at 16.0% [10.1, 22.3] flags on correct code, with diminishing returns and no plateau
Source-stated
Dong reports that the reading model and its sampling, a rulebook sentence, and the decision rule each affect the result; much of what was missed was behaviour the specification never determined, in a post hoc rating.
The reading model (confounded with its sampling), one rulebook sentence and the decision rule each move that figure, and much of what is missed is, on a post hoc rating, behaviour the specification never determined.
Source-stated
In the study checking whether reported numbers traced to their sources, all 49 inconsistent reports and none of 25 consistent fabrications were flagged; the model auditor flagged 23 consistent fabrications.
checking that reported numbers trace to their sources flagged all 49 inconsistent reports and none of 25 consistent fabrications, of which the model auditor flagged 23
Source-stated
For data, the validator and model auditor flagged overlapping but different faulty items, together 112 of 140, with a 2.9% flag rate on clean data.
on data, a documentation-derived validator and the auditor flagged overlapping but different faulty items, together 112 of 140 at a 2.9% [0.7, 5.7] flag rate on clean data
Source-stated
For scientific code, reframing the task around one step reduced flags on correct code from 68.7% to 36.0%.
on scientific code, reframing the task around one step cut flags on correct code from 68.7% to 36.0%.
Source and contributor context
Dong, Zhaohe · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 90d488bc-e906-4f1f-8134-5a9ccdadf3bc.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-19T08:46:55.383Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 9592b4b258d2e9ad5e4ca870207c13895dcca91070312e917b7cc07a7d727318. Source text hash: 91682ba86bafc67bd7ea398c86d9da7a372bf1f4ad7f0421b1a9aa275d022c63.
Revision and source-check history
Version 1 · 2026-11-19T08:46:55.383Z
arXiv cs.AI paper examines what sets the limits of automated audit
Zhaohe Dong’s paper examines a shipped arrangement in which an auditor from one vendor checks another model’s work, alongside deterministic checks. On general-purpose code with hidden tests, eight readings reached 30.0% recall while flagging 16.0% of correct code; more readings brought diminishing returns, with no plateau. The result also shifted with the reading model and its sampling, a rulebook sentence, and the decision rule. A post hoc rating found that much of what was missed was behaviour the specification never determined. In three preregistered science studies, checking whether reported numbers traced to their sources flagged all 49 inconsistent reports and none of 25 consistent fabrications; the model auditor flagged 23 fabrications. For data, a documentation-derived validator and the auditor caught overlapping but different faulty items, together flagging 112 of 140, with a 2.9% flag rate on clean data. For scientific code, framing the task around one step reduced flags on correct code from 68.7% to 36.0%. Dong’s results make the limits and variability of automated audit concrete: outcomes depended on the task and its framing, and deterministic checks and model auditors caught different errors while each had limits.
Original source
cs.AI · Article
Source details
Original source date: Sep 24, 2026 (2026-09-24).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-19T08:46:55.383Z. First observed: 2026-11-19T05:04:13.242Z. Retrieved: 2026-11-19T08:34:22.422Z.
Limits
The evidence here is the paper’s abstract; it does not provide the detailed methods or results needed to assess the studies beyond the reported figures. The abstract says that two analyses were unregistered but does not identify them in the supplied text. The abstract’s post hoc rating describes much missed behaviour as unspecified by the specification; it does not establish how that rating was conducted.
Open question
How would audit results change if specifications determined behaviour that the post hoc rating found unspecified?
Source-stated
Dong studies a shipped arrangement combining a model auditor from a different vendor with deterministic checks.
an auditor from a different vendor reads the work and deterministic checks verify what code can verify
Source-stated
For general-purpose code with hidden tests, eight readings reached 30.0% recall and flagged 16.0% of correct code, with diminishing returns and no plateau.
eight readings of the auditor reach 30.0% recall [20.0, 40.7] at 16.0% [10.1, 22.3] flags on correct code, with diminishing returns and no plateau
Source-stated
Dong reports that the reading model and its sampling, a rulebook sentence, and the decision rule each affect the result; much of what was missed was behaviour the specification never determined, in a post hoc rating.
The reading model (confounded with its sampling), one rulebook sentence and the decision rule each move that figure, and much of what is missed is, on a post hoc rating, behaviour the specification never determined.
Source-stated
In the study checking whether reported numbers traced to their sources, all 49 inconsistent reports and none of 25 consistent fabrications were flagged; the model auditor flagged 23 consistent fabrications.
checking that reported numbers trace to their sources flagged all 49 inconsistent reports and none of 25 consistent fabrications, of which the model auditor flagged 23
Source-stated
For data, the validator and model auditor flagged overlapping but different faulty items, together 112 of 140, with a 2.9% flag rate on clean data.
on data, a documentation-derived validator and the auditor flagged overlapping but different faulty items, together 112 of 140 at a 2.9% [0.7, 5.7] flag rate on clean data
Source-stated
For scientific code, reframing the task around one step reduced flags on correct code from 68.7% to 36.0%.
on scientific code, reframing the task around one step cut flags on correct code from 68.7% to 36.0%.
Source and contributor context
Dong, Zhaohe · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 90d488bc-e906-4f1f-8134-5a9ccdadf3bc.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-19T08:46:55.383Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 9592b4b258d2e9ad5e4ca870207c13895dcca91070312e917b7cc07a7d727318. Source text hash: 91682ba86bafc67bd7ea398c86d9da7a372bf1f4ad7f0421b1a9aa275d022c63.
uncertain · 2026-11-19T08:46:55.383Z · anthropic/claude-opus-5.5
Publication: 90d488bc-e906-4f1f-8134-5a9ccdadf3bc
attribution — pass: The paper is correctly attributed to Zhaohe Dong on arXiv. The title calls it a cs.AI paper; its primary category is cs.SE and it is cross-listed in cs.AI. That matches the feed and is not a misattribution.
Authors:Zhaohe Dong
summary — uncertain: The figures are faithful, but a general reader will struggle with the central terms. These include recall, eight readings, flags on correct code, deterministic checks, consistent fabrications, preregistered, post hoc and decision rule; none is explained. The summary also drops the confound between the reading model and its sampling. It omits that only 21 of the 23 auditor flags carried a sound finding, which softens the abstract's point that the model auditor erred itself. Smallest repair: explain recall and flag rate in plain words, restore the 21-of-23 detail and the confound, and note that the deterministic tier could not see consistent error.
Where both tiers ran, the deterministic one flagged no clean item and could not see consistent error; the model saw some and erred itself.
date — pass: The displayed date is 2026-09-24, which matches the v1 submission date. The later feed date in November is not used as the publication date.
[Submitted on 24 Sep 2026]
support — pass: Each numeric claim matches the abstract verbatim. The closing interpretation, that outcomes depended on task and framing and that the two kinds of check caught different errors while each had limits, is supported by the abstract. The question follows from the post hoc finding and does not introduce new facts. The limitations accurately describe what the supplied text does not show.
of which the model auditor flagged 23 (21 with a sound finding, post hoc)
media — pass: The selected media is the verified arXiv abstract page. It is the direct original source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv