# arXiv: AutoRefine adjusts code judges using workflow outcomes [Canonical reading](https://escapemeridian.com/readings/arxiv-autorefine-adjusts-code-judges-using-workflow-outcomes-77a48617) The authors of AutoRefine describe a method for calibrating a code judge—a model that assesses code changes—using outcomes from development workflows, such as tests passing or changes being abandoned. Because those outcomes are noisy, the method uses a teacher that explains the labels it keeps and abstains when it cannot explain an outcome from the judge’s input. The authors report improved agreement with workflow outcomes in their evaluations, but results differ across workflows: on older-workflow diffs, the previous judge performed better. They argue that as an agent evolves, its judge may need recalibration. Abstention can also flag outcomes the judge’s input cannot explain, suggesting that a team may need better inputs rather than a revised prompt. The abstract reports results on specific held-out datasets and diffs; it does not establish performance across other agents or workflows. Agreement with workflow labels also does not by itself establish code quality. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Nov 6, 2026 (2026-11-06). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-12T22:17:52.002Z. First observed: 2026-11-09T05:03:49.666Z. Retrieved: 2026-11-12T22:16:45.013Z. - [AutoRefine: Auto-Calibrating Code Judges from Workflow Outcomes](https://arxiv.org/abs/2611.09291) ## Limits The abstract reports evaluations on specific held-out datasets and diffs; it does not establish performance across other agents or workflows. The abstract describes workflow outcome labels as noisy, so agreement with those labels does not by itself establish code quality. ## Open question How well does AutoRefine perform as an agent evolves beyond the current and older workflow diffs reported by the authors? ## Source-stated The authors present AutoRefine as a method for rewriting a deployable judge's prompt to reproduce development-workflow outcome labels. > We present AutoRefine, a loop that rewrites a deployable judge's prompt to reproduce outcome labels - [Supporting source](https://arxiv.org/abs/2611.09291) ## Source-stated The method uses a teacher that explains labels it keeps and abstains when it cannot explain an outcome from the judge's input. > explains each label it keeps, and abstains when the outcome cannot be explained from what the judge sees. - [Supporting source](https://arxiv.org/abs/2611.09291) ## Source-stated On a held-out dataset with an execution oracle, the authors report that calibration raised chance-corrected agreement from 0.36 to 0.60. > calibration raised chance-corrected agreement (Scott's $\\pi$) from 0.36 to 0.60. - [Supporting source](https://arxiv.org/abs/2611.09291) ## Source-stated On 17 independently held-out diffs from the current workflow, recalibration raised agreement with merge/abandon outcomes; on 16 older-workflow diffs, the previous judge performed better. > on 17 independently held-out diffs from the current workflow ($\\pi$ 0.16 to 0.51); on 16 diffs from older workflows, the previous judge stayed ahead - [Supporting source](https://arxiv.org/abs/2611.09291) ## Source-stated The authors say abstention can flag changes whose outcomes the judge's input cannot explain, indicating that better inputs may be needed rather than a better prompt. > flags changes whose outcome the judge's input cannot explain, telling a team when a judge needs better inputs rather than a better prompt. - [Supporting source](https://arxiv.org/abs/2611.09291) ## Source and contributor context Shahidi, Shervin, Poon, James, Xie, Hang, Xu, Kenneth O., Håkanson, Robin, Nair, Vikram, Pillgram-Larsen, Jens, Nagappan, Nachiappan · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 6d4158f5-cd06-47ce-82fd-f31f89a85e27. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T22:17:52.002Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 5edb2882cd0fc0791d74d4c016bf0725fec094fa560785e3277bdb6af70e07b3. Source text hash: 626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342. ## Revision and source-check history ### Version 1 — 2026-11-12T22:17:52.002Z The authors of AutoRefine describe a method for calibrating a code judge—a model that assesses code changes—using outcomes from development workflows, such as tests passing or changes being abandoned. Because those outcomes are noisy, the method uses a teacher that explains the labels it keeps and abstains when it cannot explain an outcome from the judge’s input. The authors report improved agreement with workflow outcomes in their evaluations, but results differ across workflows: on older-workflow diffs, the previous judge performed better. They argue that as an agent evolves, its judge may need recalibration. Abstention can also flag outcomes the judge’s input cannot explain, suggesting that a team may need better inputs rather than a revised prompt. The abstract reports results on specific held-out datasets and diffs; it does not establish performance across other agents or workflows. Agreement with workflow labels also does not by itself establish code quality. ## Original source arXiv cs.AI · Article Original source date: Nov 6, 2026 (2026-11-06). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-12T22:17:52.002Z. First observed: 2026-11-09T05:03:49.666Z. Retrieved: 2026-11-12T22:16:45.013Z. - [AutoRefine: Auto-Calibrating Code Judges from Workflow Outcomes](https://arxiv.org/abs/2611.09291) ## Limits The abstract reports evaluations on specific held-out datasets and diffs; it does not establish performance across other agents or workflows. The abstract describes workflow outcome labels as noisy, so agreement with those labels does not by itself establish code quality. ## Open question How well does AutoRefine perform as an agent evolves beyond the current and older workflow diffs reported by the authors? ## Source-stated The authors present AutoRefine as a method for rewriting a deployable judge's prompt to reproduce development-workflow outcome labels. > We present AutoRefine, a loop that rewrites a deployable judge's prompt to reproduce outcome labels - [Supporting source](https://arxiv.org/abs/2611.09291) ## Source-stated The method uses a teacher that explains labels it keeps and abstains when it cannot explain an outcome from the judge's input. > explains each label it keeps, and abstains when the outcome cannot be explained from what the judge sees. - [Supporting source](https://arxiv.org/abs/2611.09291) ## Source-stated On a held-out dataset with an execution oracle, the authors report that calibration raised chance-corrected agreement from 0.36 to 0.60. > calibration raised chance-corrected agreement (Scott's $\\pi$) from 0.36 to 0.60. - [Supporting source](https://arxiv.org/abs/2611.09291) ## Source-stated On 17 independently held-out diffs from the current workflow, recalibration raised agreement with merge/abandon outcomes; on 16 older-workflow diffs, the previous judge performed better. > on 17 independently held-out diffs from the current workflow ($\\pi$ 0.16 to 0.51); on 16 diffs from older workflows, the previous judge stayed ahead - [Supporting source](https://arxiv.org/abs/2611.09291) ## Source-stated The authors say abstention can flag changes whose outcomes the judge's input cannot explain, indicating that better inputs may be needed rather than a better prompt. > flags changes whose outcome the judge's input cannot explain, telling a team when a judge needs better inputs rather than a better prompt. - [Supporting source](https://arxiv.org/abs/2611.09291) ## Source and contributor context Shahidi, Shervin, Poon, James, Xie, Hang, Xu, Kenneth O., Håkanson, Robin, Nair, Vikram, Pillgram-Larsen, Jens, Nagappan, Nachiappan · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 6d4158f5-cd06-47ce-82fd-f31f89a85e27. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-12T22:17:52.002Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 5edb2882cd0fc0791d74d4c016bf0725fec094fa560785e3277bdb6af70e07b3. Source text hash: 626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342. ### uncertain — 2026-11-12T22:17:52.002Z — anthropic/claude-opus-5.5 Publication: 6d4158f5-cd06-47ce-82fd-f31f89a85e27 attribution — pass: The reading consistently attributes the method, results and recalibration argument to the AutoRefine authors on arXiv. It does not present their claims as established fact or as Escape Meridian's view. > We present AutoRefine, a loop that rewrites a deployable judge's prompt to reproduce outcome labels summary — uncertain: The core idea is mostly clear, but several central terms and details are not readable on their own. 'Teacher', 'diffs' and 'held-out' are unexplained. The summary never says the method rewrites the judge's prompt, yet later mentions 'a revised prompt'. It omits why judges matter here: some code changes have no test oracle. It also omits the noise result (25% to 20%). It says agreement improved 'with workflow outcomes', but the 0.36 to 0.60 result was measured on a dataset with an execution oracle. Smallest repairs: define the teacher, explain 'diffs', state that the prompt is rewritten, and separate the two evaluations. > LLM judges increasingly serve as quality gates for code changes that have no test oracle date — pass: evidence.originalPublished is 2026-11-06, which matches the arXiv first-submission date for v1. The feed date of 2026-11-09 is correctly not used. > \[Submitted on 6 Nov 2026\] support — pass: Each claim matches a verbatim excerpt from the abstract. The older-workflow caveat is preserved. The question about performance as the agent evolves follows from the authors' own mixed result. The limitations avoid inventing benefits or harms. The caveat that agreement with labels does not establish code quality is a reasonable scoping note, grounded in the abstract's statement that the labels are noisy. > on 16 diffs from older workflows, the previous judge stayed ahead, illustrating the need for recalibration as the agent evolves media — pass: The selected media is the verified, direct arXiv abstract page for this paper, which is the most direct way to access the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/77a4861780d7ee5518dd5404aae6038795b5c7c7a0c550044db05e9fa1daad65/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-autorefine-adjusts-code-judges-using-workflow-outcomes-77a48617.json)