arXiv paper reports blind spots in medical AI evaluation rubrics

arXiv paper reports blind spots in medical AI evaluation rubrics

In “When Rubrics Fail: Hallucinations Reveal Blind Spots in Medical AI Evaluation,” Griffin Farrow and coauthors report that rubric scores can miss clinically relevant errors in LLM responses. Rubric-based evaluation uses criteria to assess responses; the authors tested whether its scores reflect errors that matter clinically. In a controlled setting using MedHallu, they found that more specific rubrics better distinguished correct from hallucinated responses. Across HealthBench, HealthBench Professional, and LiveMedBench, they report that clinically relevant hallucinations were often missed, leaving scores unchanged. Rubrics worked best when they explicitly checked facts, and were less effective for unexpected errors they did not anticipate. The authors say a preliminary retrieval-based factuality check recovered some errors missed by rubrics. They argue that rubric scores alone are insufficient to establish clinical reliability. The reported results concern evaluations in the study; they do not establish how reliably any particular system performs in clinical use. Open question: How well would retrieval-based factuality checks identify clinically relevant errors beyond the cases tested in this study?

Publication: 0e077a92-f9b9-4dbb-83be-25d5b8ea9482

attribution — pass: The paper, its title, and the claims are correctly attributed to Griffin Farrow and coauthors on arXiv. The findings are framed as the authors' reports and arguments, not as established fact.

Authors:Griffin Farrow, Lily Sijia Li, Jack Johnson, Tingyan Wang, Philip Torr, William Bolton, Fabio J. Fehr

summary — uncertain: Central terms are unexplained for a general reader: LLM, hallucination, and the fact that MedHallu, HealthBench, HealthBench Professional and LiveMedBench are medical test sets (benchmarks). The draft also omits a key method. The hallucinations were errors the authors deliberately injected, using a clinician-validated pipeline and a taxonomy of error types. Writing just 'clinically relevant hallucinations' blurs this condition. Smallest repair: add plain glosses or term-library tooltips for these terms, and say the errors were clinician-validated injected errors in matched responses.

a clinician-validated error-injection pipeline that creates matched correct and error-injected responses

date — pass: originalPublished is 2026-09-11, which matches the v1 submission date. The later v2 revision and the feed date are correctly not used.

Submitted on 11 Sep 2026 (v1), last revised 27 Oct 2026 (this version, v2)

support — uncertain: Most claims match the abstract closely. However, the summary and claim 2 say the hallucinations were 'often missed'. The source says they 'are missed', and the hedge 'often' applies only to scores staying unchanged. The paraphrase shifts the scope slightly. Dropping 'our' also obscures that these were the authors' injected errors. The limitations and the open question are reasonable and not leading.

our clinically relevant hallucinations are missed by rubrics, often leaving scores unchanged

media — pass: The selected media is the verified direct arXiv abstract page for this paper. It gives readers the easiest access to the source.