# arXiv cs.AI paper examines what sets the limits of automated audit

[Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-paper-examines-what-sets-the-limits-of-automated-7be4fc5a)

Zhaohe Dong’s arXiv paper examines a shipped arrangement in which an auditor from one vendor checks another model’s work, alongside deterministic checks. On general-purpose code with hidden tests, eight readings reached 30.0% recall while flagging 16.0% of correct code; more readings brought diminishing returns, with no plateau. The result also shifted with the reading model and its sampling, a rulebook sentence, and the decision rule. A post hoc rating found that much of what was missed was behaviour the specification never determined. In three preregistered science studies, checking whether reported numbers traced to their sources flagged all 49 inconsistent reports and none of 25 consistent fabrications; the model auditor flagged 23 fabrications. For data, a documentation-derived validator and the auditor caught overlapping but different faulty items, together flagging 112 of 140, with a 2.9% flag rate on clean data. For scientific code, framing the task around one step reduced flags on correct code from 68.7% to 36.0%. Dong’s results make the limits and variability of automated audit concrete: outcomes depended on the task and its framing, and deterministic checks and model auditors caught different errors while each had limits.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv cs.AI · Article

Original source date: Sep 24, 2026 (2026-09-24).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-19T08:46:55.383Z. First observed: 2026-11-19T05:04:13.242Z. Retrieved: 2026-11-19T08:34:22.422Z.

- [What Moves the Ceiling on Automated Audit: from General Code to Scientific Code, Results and Data](https://arxiv.org/abs/2611.23250)

## Limits

The evidence here is the paper’s abstract; it does not provide the detailed methods or results needed to assess the studies beyond the reported figures. The abstract says that two analyses were unregistered but does not identify them in the supplied text. The abstract’s post hoc rating describes much missed behaviour as unspecified by the specification; it does not establish how that rating was conducted.


## Open question

How would audit results change if specifications determined behaviour that the post hoc rating found unspecified?


## Source-stated

Dong studies a shipped arrangement combining a model auditor from a different vendor with deterministic checks.

> an auditor from a different vendor reads the work and deterministic checks verify what code can verify

- [Supporting source](https://arxiv.org/abs/2611.23250)

## Source-stated

For general-purpose code with hidden tests, eight readings reached 30.0% recall and flagged 16.0% of correct code, with diminishing returns and no plateau.

> eight readings of the auditor reach 30.0% recall \[20.0, 40.7\] at 16.0% \[10.1, 22.3\] flags on correct code, with diminishing returns and no plateau

- [Supporting source](https://arxiv.org/abs/2611.23250)

## Source-stated

Dong reports that the reading model and its sampling, a rulebook sentence, and the decision rule each affect the result; much of what was missed was behaviour the specification never determined, in a post hoc rating.

> The reading model (confounded with its sampling), one rulebook sentence and the decision rule each move that figure, and much of what is missed is, on a post hoc rating, behaviour the specification never determined.

- [Supporting source](https://arxiv.org/abs/2611.23250)

## Source-stated

In the study checking whether reported numbers traced to their sources, all 49 inconsistent reports and none of 25 consistent fabrications were flagged; the model auditor flagged 23 consistent fabrications.

> checking that reported numbers trace to their sources flagged all 49 inconsistent reports and none of 25 consistent fabrications, of which the model auditor flagged 23

- [Supporting source](https://arxiv.org/abs/2611.23250)

## Source-stated

For data, the validator and model auditor flagged overlapping but different faulty items, together 112 of 140, with a 2.9% flag rate on clean data.

> on data, a documentation-derived validator and the auditor flagged overlapping but different faulty items, together 112 of 140 at a 2.9% \[0.7, 5.7\] flag rate on clean data

- [Supporting source](https://arxiv.org/abs/2611.23250)

## Source-stated

For scientific code, reframing the task around one step reduced flags on correct code from 68.7% to 36.0%.

> on scientific code, reframing the task around one step cut flags on correct code from 68.7% to 36.0%.

- [Supporting source](https://arxiv.org/abs/2611.23250)

## Source and contributor context

Dong, Zhaohe · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: 90d488bc-e906-4f1f-8134-5a9ccdadf3bc.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-19T08:46:55.383Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 9592b4b258d2e9ad5e4ca870207c13895dcca91070312e917b7cc07a7d727318. Source text hash: 91682ba86bafc67bd7ea398c86d9da7a372bf1f4ad7f0421b1a9aa275d022c63.


## Revision and source-check history

### Version 1 — 2026-11-19T08:46:55.383Z

Zhaohe Dong’s arXiv paper examines a shipped arrangement in which an auditor from one vendor checks another model’s work, alongside deterministic checks. On general-purpose code with hidden tests, eight readings reached 30.0% recall while flagging 16.0% of correct code; more readings brought diminishing returns, with no plateau. The result also shifted with the reading model and its sampling, a rulebook sentence, and the decision rule. A post hoc rating found that much of what was missed was behaviour the specification never determined. In three preregistered science studies, checking whether reported numbers traced to their sources flagged all 49 inconsistent reports and none of 25 consistent fabrications; the model auditor flagged 23 fabrications. For data, a documentation-derived validator and the auditor caught overlapping but different faulty items, together flagging 112 of 140, with a 2.9% flag rate on clean data. For scientific code, framing the task around one step reduced flags on correct code from 68.7% to 36.0%. Dong’s results make the limits and variability of automated audit concrete: outcomes depended on the task and its framing, and deterministic checks and model auditors caught different errors while each had limits.

## Original source

arXiv cs.AI · Article

Original source date: Sep 24, 2026 (2026-09-24).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-19T08:46:55.383Z. First observed: 2026-11-19T05:04:13.242Z. Retrieved: 2026-11-19T08:34:22.422Z.

- [What Moves the Ceiling on Automated Audit: from General Code to Scientific Code, Results and Data](https://arxiv.org/abs/2611.23250)

## Limits

The evidence here is the paper’s abstract; it does not provide the detailed methods or results needed to assess the studies beyond the reported figures. The abstract says that two analyses were unregistered but does not identify them in the supplied text. The abstract’s post hoc rating describes much missed behaviour as unspecified by the specification; it does not establish how that rating was conducted.


## Open question

How would audit results change if specifications determined behaviour that the post hoc rating found unspecified?


## Source-stated

Dong studies a shipped arrangement combining a model auditor from a different vendor with deterministic checks.

> an auditor from a different vendor reads the work and deterministic checks verify what code can verify

- [Supporting source](https://arxiv.org/abs/2611.23250)

## Source-stated

For general-purpose code with hidden tests, eight readings reached 30.0% recall and flagged 16.0% of correct code, with diminishing returns and no plateau.

> eight readings of the auditor reach 30.0% recall \[20.0, 40.7\] at 16.0% \[10.1, 22.3\] flags on correct code, with diminishing returns and no plateau

- [Supporting source](https://arxiv.org/abs/2611.23250)

## Source-stated

Dong reports that the reading model and its sampling, a rulebook sentence, and the decision rule each affect the result; much of what was missed was behaviour the specification never determined, in a post hoc rating.

> The reading model (confounded with its sampling), one rulebook sentence and the decision rule each move that figure, and much of what is missed is, on a post hoc rating, behaviour the specification never determined.

- [Supporting source](https://arxiv.org/abs/2611.23250)

## Source-stated

In the study checking whether reported numbers traced to their sources, all 49 inconsistent reports and none of 25 consistent fabrications were flagged; the model auditor flagged 23 consistent fabrications.

> checking that reported numbers trace to their sources flagged all 49 inconsistent reports and none of 25 consistent fabrications, of which the model auditor flagged 23

- [Supporting source](https://arxiv.org/abs/2611.23250)

## Source-stated

For data, the validator and model auditor flagged overlapping but different faulty items, together 112 of 140, with a 2.9% flag rate on clean data.

> on data, a documentation-derived validator and the auditor flagged overlapping but different faulty items, together 112 of 140 at a 2.9% \[0.7, 5.7\] flag rate on clean data

- [Supporting source](https://arxiv.org/abs/2611.23250)

## Source-stated

For scientific code, reframing the task around one step reduced flags on correct code from 68.7% to 36.0%.

> on scientific code, reframing the task around one step cut flags on correct code from 68.7% to 36.0%.

- [Supporting source](https://arxiv.org/abs/2611.23250)

## Source and contributor context

Dong, Zhaohe · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: 90d488bc-e906-4f1f-8134-5a9ccdadf3bc.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-19T08:46:55.383Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 9592b4b258d2e9ad5e4ca870207c13895dcca91070312e917b7cc07a7d727318. Source text hash: 91682ba86bafc67bd7ea398c86d9da7a372bf1f4ad7f0421b1a9aa275d022c63.


### uncertain — 2026-11-19T08:46:55.383Z — anthropic/claude-opus-5.5

Publication: 90d488bc-e906-4f1f-8134-5a9ccdadf3bc

attribution — pass: The paper is correctly attributed to Zhaohe Dong on arXiv. The title calls it a cs.AI paper; its primary category is cs.SE and it is cross-listed in cs.AI. That matches the feed and is not a misattribution.

> Authors:Zhaohe Dong

summary — uncertain: The figures are faithful, but a general reader will struggle with the central terms. These include recall, eight readings, flags on correct code, deterministic checks, consistent fabrications, preregistered, post hoc and decision rule; none is explained. The summary also drops the confound between the reading model and its sampling. It omits that only 21 of the 23 auditor flags carried a sound finding, which softens the abstract's point that the model auditor erred itself. Smallest repair: explain recall and flag rate in plain words, restore the 21-of-23 detail and the confound, and note that the deterministic tier could not see consistent error.

> Where both tiers ran, the deterministic one flagged no clean item and could not see consistent error; the model saw some and erred itself.

date — pass: The displayed date is 2026-09-24, which matches the v1 submission date. The later feed date in November is not used as the publication date.

> \[Submitted on 24 Sep 2026\]

support — pass: Each numeric claim matches the abstract verbatim. The closing interpretation, that outcomes depended on task and framing and that the two kinds of check caught different errors while each had limits, is supported by the abstract. The question follows from the post hoc finding and does not introduce new facts. The limitations accurately describe what the supplied text does not show.

> of which the model auditor flagged 23 (21 with a sound finding, post hoc)

media — pass: The selected media is the verified arXiv abstract page. It is the direct original source.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/7be4fc5a96d5af47d0da27aca07dca6e185fb6e97a9787edfc52eb2681a7e416/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-paper-examines-what-sets-the-limits-of-automated-7be4fc5a.json)
