arXiv cs.AI: George Koomullil on measurement when AI readers disagree

arXiv cs.AI: George Koomullil on measurement when AI readers disagree

George Koomullil studies how to estimate monetary totals from administrative text when people and AI systems assign labels differently. In a testbed of grant-purpose text from US tax filings, the paper compares machine predictions, human review and ways of selecting records for review. Its central tension is practical: predictions can help direct limited review, but disagreement among human readers can shift the estimated total substantially. The paper reports that independent readers shifted the estimate by $3.93B, more than twice the within-reader sampling interval, and that institution-level intervals under-covered at every size. In a replay charging for every human label, a two-phase review policy whose calibrator is fitted on its own first phase reduced RMSE, an estimation-error measure, by 28% at a budget of 100 texts compared with dollar-proportional sampling. But the authors say the error control of two rules for accepting machine labels in deployment is not established; one count-based rule accepted no text under a dollar-drawn sample. The paper reports both potential gains from using predictions to guide human checks and limits that matter when the target is a monetary total. It reruns headline results under each reader, treating disagreement among people as part of the measurement problem rather than treating one reading as definitive.

Publication: 3c8a24e3-f128-4900-86d9-c79a70418ede

attribution — pass: The paper is correctly attributed to George Koomullil (Ascendr, PBC) on arXiv cs.AI, and findings are framed as the paper's reports. One small note: the summary says 'the authors say', while the listing shows a single author. This does not misattribute any claim.

Authors:George Koomullil (Ascendr, PBC)

summary — uncertain: The summary is mostly faithful, but it leaves out a headline result: once rectified, 8 predictors whose own totals ranged from $11.2B to $15.0B agree to within $0.4B. That result helps balance the reader-disagreement finding. It also omits that pure Neyman allocation fails with an uncalibrated predictor. 'Under-covered', 'calibrator', 'within-reader sampling interval' and 'dollar-proportional sampling' are not explained for general readers. Smallest repair: add the rectification-agreement sentence and briefly gloss under-coverage (the intervals contained the true value less often than stated) and calibrator, or add tooltips for these terms.

8 predictors whose own totals span \$11.2-15.0B agree to within \$0.4B once rectified

date — pass: The originalPublished value of 2026-11-09 matches the arXiv v1 submission date. The feed date of 11 Nov is later and was correctly not used.

[Submitted on 9 Nov 2026]

support — pass: Each figure is quoted verbatim from the abstract: $3.93B and 2.2 times, the 28% RMSE reduction at 100 texts, the non-established deployment error control, and the count-based rule accepting no text. The question follows directly from a limitation the source states. One phrase, 'one count-based rule', slightly implies that the count-based rule is one of the two named rules, which the abstract leaves ambiguous. This is a minor issue, not a contradiction.

their error control in deployment is not established, and the count-based rule accepts no text under a dollar-drawn sample.

media — pass: The selected media is the verified arXiv abstract page for this paper. It is the direct original source.