{"id":"77a4861780d7ee5518dd5404aae6038795b5c7c7a0c550044db05e9fa1daad65","title":"arXiv: AutoRefine adjusts code judges using workflow outcomes","argument":"The authors of AutoRefine describe a method for calibrating a code judge—a model that assesses code changes—using outcomes from development workflows, such as tests passing or changes being abandoned. Because those outcomes are noisy, the method uses a teacher that explains the labels it keeps and abstains when it cannot explain an outcome from the judge’s input. The authors report improved agreement with workflow outcomes in their evaluations, but results differ across workflows: on older-workflow diffs, the previous judge performed better. They argue that as an agent evolves, its judge may need recalibration. Abstention can also flag outcomes the judge’s input cannot explain, suggesting that a team may need better inputs rather than a revised prompt. The abstract reports results on specific held-out datasets and diffs; it does not establish performance across other agents or workflows. Agreement with workflow labels also does not by itself establish code quality.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract reports evaluations on specific held-out datasets and diffs; it does not establish performance across other agents or workflows. The abstract describes workflow outcome labels as noisy, so agreement with those labels does not by itself establish code quality.","speaker":{"name":"Shahidi, Shervin, Poon, James, Xie, Hang, Xu, Kenneth O., Håkanson, Robin, Nair, Vikram, Pillgram-Larsen, Jens, Nagappan, Nachiappan","roleAtSource":"Article author"},"source":{"title":"AutoRefine: Auto-Calibrating Code Judges from Workflow Outcomes","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.09291","publishedAt":"2026-11-06","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors present AutoRefine as a method for rewriting a deployable judge's prompt to reproduce development-workflow outcome labels.","excerpt":"We present AutoRefine, a loop that rewrites a deployable judge's prompt to reproduce outcome labels","url":"https://arxiv.org/abs/2611.09291","relationship":"Source-stated"},{"text":"The method uses a teacher that explains labels it keeps and abstains when it cannot explain an outcome from the judge's input.","excerpt":"explains each label it keeps, and abstains when the outcome cannot be explained from what the judge sees.","url":"https://arxiv.org/abs/2611.09291","relationship":"Source-stated"},{"text":"On a held-out dataset with an execution oracle, the authors report that calibration raised chance-corrected agreement from 0.36 to 0.60.","excerpt":"calibration raised chance-corrected agreement (Scott's $\\pi$) from 0.36 to 0.60.","url":"https://arxiv.org/abs/2611.09291","relationship":"Source-stated"},{"text":"On 17 independently held-out diffs from the current workflow, recalibration raised agreement with merge/abandon outcomes; on 16 older-workflow diffs, the previous judge performed better.","excerpt":"on 17 independently held-out diffs from the current workflow ($\\pi$ 0.16 to 0.51); on 16 diffs from older workflows, the previous judge stayed ahead","url":"https://arxiv.org/abs/2611.09291","relationship":"Source-stated"},{"text":"The authors say abstention can flag changes whose outcomes the judge's input cannot explain, indicating that better inputs may be needed rather than a better prompt.","excerpt":"flags changes whose outcome the judge's input cannot explain, telling a team when a judge needs better inputs rather than a better prompt.","url":"https://arxiv.org/abs/2611.09291","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.09291","checkedAt":"2026-11-12T22:48:10.385Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well does AutoRefine perform as an agent evolves beyond the current and older workflow diffs reported by the authors?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv: AutoRefine adjusts code judges using workflow outcomes","summary":"The authors of AutoRefine describe a method for calibrating a code judge—a model that assesses code changes—using outcomes from development workflows, such as tests passing or changes being abandoned. Because those outcomes are noisy, the method uses a teacher that explains the labels it keeps and abstains when it cannot explain an outcome from the judge’s input. The authors report improved agreement with workflow outcomes in their evaluations, but results differ across workflows: on older-workflow diffs, the previous judge performed better. They argue that as an agent evolves, its judge may need recalibration. Abstention can also flag outcomes the judge’s input cannot explain, suggesting that a team may need better inputs rather than a revised prompt. The abstract reports results on specific held-out datasets and diffs; it does not establish performance across other agents or workflows. Agreement with workflow labels also does not by itself establish code quality.","question":"How well does AutoRefine perform as an agent evolves beyond the current and older workflow diffs reported by the authors?","claims":[{"text":"The authors present AutoRefine as a method for rewriting a deployable judge's prompt to reproduce development-workflow outcome labels.","excerpt":"We present AutoRefine, a loop that rewrites a deployable judge's prompt to reproduce outcome labels","support":"source-stated"},{"text":"The method uses a teacher that explains labels it keeps and abstains when it cannot explain an outcome from the judge's input.","excerpt":"explains each label it keeps, and abstains when the outcome cannot be explained from what the judge sees.","support":"source-stated"},{"text":"On a held-out dataset with an execution oracle, the authors report that calibration raised chance-corrected agreement from 0.36 to 0.60.","excerpt":"calibration raised chance-corrected agreement (Scott's $\\pi$) from 0.36 to 0.60.","support":"source-stated"},{"text":"On 17 independently held-out diffs from the current workflow, recalibration raised agreement with merge/abandon outcomes; on 16 older-workflow diffs, the previous judge performed better.","excerpt":"on 17 independently held-out diffs from the current workflow ($\\pi$ 0.16 to 0.51); on 16 diffs from older workflows, the previous judge stayed ahead","support":"source-stated"},{"text":"The authors say abstention can flag changes whose outcomes the judge's input cannot explain, indicating that better inputs may be needed rather than a better prompt.","excerpt":"flags changes whose outcome the judge's input cannot explain, telling a team when a judge needs better inputs rather than a better prompt.","support":"source-stated"}],"limitations":["The abstract reports evaluations on specific held-out datasets and diffs; it does not establish performance across other agents or workflows.","The abstract describes workflow outcome labels as noisy, so agreement with those labels does not by itself establish code quality."],"mediaId":"ea813f828db1bd3bc39bdd3e675a288e84501b558de29189f645ae9d66ff7c41","correctionReason":null,"id":"77a4861780d7ee5518dd5404aae6038795b5c7c7a0c550044db05e9fa1daad65","publicationId":"6d4158f5-cd06-47ce-82fd-f31f89a85e27","version":1,"sourceUrl":"https://arxiv.org/abs/2611.09291","sourceTitle":"AutoRefine: Auto-Calibrating Code Judges from Workflow Outcomes","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-06","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-09T05:03:49.666Z","createdAt":"2026-11-12T22:17:52.002Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"ea813f828db1bd3bc39bdd3e675a288e84501b558de29189f645ae9d66ff7c41","kind":"article","url":"https://arxiv.org/abs/2611.09291","sourceUrl":"https://arxiv.org/abs/2611.09291","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T22:16:45.142Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.09291","checkedAt":"2026-11-12T22:48:10.385Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"ea813f828db1bd3bc39bdd3e675a288e84501b558de29189f645ae9d66ff7c41","confidence":0.63,"provider":"typesafe","model":"jev-1.13.0","inputHash":"62ab0d84631d6e94e2e8075d69d272da45109e5a2c45b88a1f9c11d02fd1237c","decidedAt":"2026-11-12T22:16:46.208Z"},"provenance":{"textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observationId":"5edb2882cd0fc0791d74d4c016bf0725fec094fa560785e3277bdb6af70e07b3","retrievedAt":"2026-11-12T22:16:45.013Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"d67511cb-369b-4812-8372-136ed093a7bf"},"reviewer":{"model":"gpt-6-luna","runId":"6e67a227-6b63-4c2a-8983-7c2dbad7c95e"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-12T22:17:52.002Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Shahidi, Shervin","url":null,"kind":"unknown"},{"name":"Poon, James","url":null,"kind":"unknown"},{"name":"Xie, Hang","url":null,"kind":"unknown"},{"name":"Xu, Kenneth O.","url":null,"kind":"unknown"},{"name":"Håkanson, Robin","url":null,"kind":"unknown"},{"name":"Nair, Vikram","url":null,"kind":"unknown"},{"name":"Pillgram-Larsen, Jens","url":null,"kind":"unknown"},{"name":"Nagappan, Nachiappan","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Shahidi, Shervin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Shahidi, Shervin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09291","locator":"head/meta[name=author|citation_author]"},"assertionId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T22:48:10.385Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41","sourceUrl":"https://arxiv.org/abs/2611.09291","contentSha256":"7b3355425cb7e4036950d0a4305d46fb5dd9ccc882b81aacf350541bc09e73a1","textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observedAt":"2026-11-12T22:48:10.385Z"}},{"actor":{"name":"Poon, James","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Poon, James","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09291","locator":"head/meta[name=author|citation_author]"},"assertionId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T22:48:10.385Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41","sourceUrl":"https://arxiv.org/abs/2611.09291","contentSha256":"7b3355425cb7e4036950d0a4305d46fb5dd9ccc882b81aacf350541bc09e73a1","textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observedAt":"2026-11-12T22:48:10.385Z"}},{"actor":{"name":"Xie, Hang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Xie, Hang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09291","locator":"head/meta[name=author|citation_author]"},"assertionId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T22:48:10.385Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41","sourceUrl":"https://arxiv.org/abs/2611.09291","contentSha256":"7b3355425cb7e4036950d0a4305d46fb5dd9ccc882b81aacf350541bc09e73a1","textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observedAt":"2026-11-12T22:48:10.385Z"}},{"actor":{"name":"Xu, Kenneth O.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Xu, Kenneth O.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09291","locator":"head/meta[name=author|citation_author]"},"assertionId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T22:48:10.385Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41","sourceUrl":"https://arxiv.org/abs/2611.09291","contentSha256":"7b3355425cb7e4036950d0a4305d46fb5dd9ccc882b81aacf350541bc09e73a1","textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observedAt":"2026-11-12T22:48:10.385Z"}},{"actor":{"name":"Håkanson, Robin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Håkanson, Robin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09291","locator":"head/meta[name=author|citation_author]"},"assertionId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T22:48:10.385Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41","sourceUrl":"https://arxiv.org/abs/2611.09291","contentSha256":"7b3355425cb7e4036950d0a4305d46fb5dd9ccc882b81aacf350541bc09e73a1","textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observedAt":"2026-11-12T22:48:10.385Z"}},{"actor":{"name":"Nair, Vikram","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Nair, Vikram","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09291","locator":"head/meta[name=author|citation_author]"},"assertionId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T22:48:10.385Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41","sourceUrl":"https://arxiv.org/abs/2611.09291","contentSha256":"7b3355425cb7e4036950d0a4305d46fb5dd9ccc882b81aacf350541bc09e73a1","textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observedAt":"2026-11-12T22:48:10.385Z"}},{"actor":{"name":"Pillgram-Larsen, Jens","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Pillgram-Larsen, Jens","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09291","locator":"head/meta[name=author|citation_author]"},"assertionId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T22:48:10.385Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41","sourceUrl":"https://arxiv.org/abs/2611.09291","contentSha256":"7b3355425cb7e4036950d0a4305d46fb5dd9ccc882b81aacf350541bc09e73a1","textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observedAt":"2026-11-12T22:48:10.385Z"}},{"actor":{"name":"Nagappan, Nachiappan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Nagappan, Nachiappan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09291","locator":"head/meta[name=author|citation_author]"},"assertionId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T22:48:10.385Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41","sourceUrl":"https://arxiv.org/abs/2611.09291","contentSha256":"7b3355425cb7e4036950d0a4305d46fb5dd9ccc882b81aacf350541bc09e73a1","textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observedAt":"2026-11-12T22:48:10.385Z"}}],"participationCoverage":{"total":8,"returned":8,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.09291","snapshotId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41","contentSha256":"7b3355425cb7e4036950d0a4305d46fb5dd9ccc882b81aacf350541bc09e73a1","textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observedAt":"2026-11-12T22:48:10.385Z","basis":"publisher-page-metadata","sourceLinks":47,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/77a4861780d7ee5518dd5404aae6038795b5c7c7a0c550044db05e9fa1daad65/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-autorefine-adjusts-code-judges-using-workflow-outcomes-77a48617","formats":{"html":"/readings/arxiv-autorefine-adjusts-code-judges-using-workflow-outcomes-77a48617","markdown":"/readings/arxiv-autorefine-adjusts-code-judges-using-workflow-outcomes-77a48617.md","json":"/readings/arxiv-autorefine-adjusts-code-judges-using-workflow-outcomes-77a48617.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"77a4861780d7ee5518dd5404aae6038795b5c7c7a0c550044db05e9fa1daad65","revision":"6d4158f5-cd06-47ce-82fd-f31f89a85e27","revisionKind":"publication-id","title":"arXiv: AutoRefine adjusts code judges using workflow outcomes","url":"https://escapemeridian.com/readings/arxiv-autorefine-adjusts-code-judges-using-workflow-outcomes-77a48617"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.09291","revision":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","revisionKind":"sha256-source-text","title":"AutoRefine: Auto-Calibrating Code Judges from Workflow Outcomes","url":"https://arxiv.org/abs/2611.09291"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"77a4861780d7ee5518dd5404aae6038795b5c7c7a0c550044db05e9fa1daad65","title":"arXiv: AutoRefine adjusts code judges using workflow outcomes","argument":"The authors of AutoRefine describe a method for calibrating a code judge—a model that assesses code changes—using outcomes from development workflows, such as tests passing or changes being abandoned. Because those outcomes are noisy, the method uses a teacher that explains the labels it keeps and abstains when it cannot explain an outcome from the judge’s input. The authors report improved agreement with workflow outcomes in their evaluations, but results differ across workflows: on older-workflow diffs, the previous judge performed better. They argue that as an agent evolves, its judge may need recalibration. Abstention can also flag outcomes the judge’s input cannot explain, suggesting that a team may need better inputs rather than a revised prompt. The abstract reports results on specific held-out datasets and diffs; it does not establish performance across other agents or workflows. Agreement with workflow labels also does not by itself establish code quality.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract reports evaluations on specific held-out datasets and diffs; it does not establish performance across other agents or workflows. The abstract describes workflow outcome labels as noisy, so agreement with those labels does not by itself establish code quality.","speaker":{"name":"Shahidi, Shervin, Poon, James, Xie, Hang, Xu, Kenneth O., Håkanson, Robin, Nair, Vikram, Pillgram-Larsen, Jens, Nagappan, Nachiappan","roleAtSource":"Article author"},"source":{"title":"AutoRefine: Auto-Calibrating Code Judges from Workflow Outcomes","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.09291","publishedAt":"2026-11-06","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors present AutoRefine as a method for rewriting a deployable judge's prompt to reproduce development-workflow outcome labels.","excerpt":"We present AutoRefine, a loop that rewrites a deployable judge's prompt to reproduce outcome labels","url":"https://arxiv.org/abs/2611.09291","relationship":"Source-stated"},{"text":"The method uses a teacher that explains labels it keeps and abstains when it cannot explain an outcome from the judge's input.","excerpt":"explains each label it keeps, and abstains when the outcome cannot be explained from what the judge sees.","url":"https://arxiv.org/abs/2611.09291","relationship":"Source-stated"},{"text":"On a held-out dataset with an execution oracle, the authors report that calibration raised chance-corrected agreement from 0.36 to 0.60.","excerpt":"calibration raised chance-corrected agreement (Scott's $\\pi$) from 0.36 to 0.60.","url":"https://arxiv.org/abs/2611.09291","relationship":"Source-stated"},{"text":"On 17 independently held-out diffs from the current workflow, recalibration raised agreement with merge/abandon outcomes; on 16 older-workflow diffs, the previous judge performed better.","excerpt":"on 17 independently held-out diffs from the current workflow ($\\pi$ 0.16 to 0.51); on 16 diffs from older workflows, the previous judge stayed ahead","url":"https://arxiv.org/abs/2611.09291","relationship":"Source-stated"},{"text":"The authors say abstention can flag changes whose outcomes the judge's input cannot explain, indicating that better inputs may be needed rather than a better prompt.","excerpt":"flags changes whose outcome the judge's input cannot explain, telling a team when a judge needs better inputs rather than a better prompt.","url":"https://arxiv.org/abs/2611.09291","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.09291","checkedAt":"2026-11-12T22:48:10.385Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well does AutoRefine perform as an agent evolves beyond the current and older workflow diffs reported by the authors?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv: AutoRefine adjusts code judges using workflow outcomes","summary":"The authors of AutoRefine describe a method for calibrating a code judge—a model that assesses code changes—using outcomes from development workflows, such as tests passing or changes being abandoned. Because those outcomes are noisy, the method uses a teacher that explains the labels it keeps and abstains when it cannot explain an outcome from the judge’s input. The authors report improved agreement with workflow outcomes in their evaluations, but results differ across workflows: on older-workflow diffs, the previous judge performed better. They argue that as an agent evolves, its judge may need recalibration. Abstention can also flag outcomes the judge’s input cannot explain, suggesting that a team may need better inputs rather than a revised prompt. The abstract reports results on specific held-out datasets and diffs; it does not establish performance across other agents or workflows. Agreement with workflow labels also does not by itself establish code quality.","question":"How well does AutoRefine perform as an agent evolves beyond the current and older workflow diffs reported by the authors?","claims":[{"text":"The authors present AutoRefine as a method for rewriting a deployable judge's prompt to reproduce development-workflow outcome labels.","excerpt":"We present AutoRefine, a loop that rewrites a deployable judge's prompt to reproduce outcome labels","support":"source-stated"},{"text":"The method uses a teacher that explains labels it keeps and abstains when it cannot explain an outcome from the judge's input.","excerpt":"explains each label it keeps, and abstains when the outcome cannot be explained from what the judge sees.","support":"source-stated"},{"text":"On a held-out dataset with an execution oracle, the authors report that calibration raised chance-corrected agreement from 0.36 to 0.60.","excerpt":"calibration raised chance-corrected agreement (Scott's $\\pi$) from 0.36 to 0.60.","support":"source-stated"},{"text":"On 17 independently held-out diffs from the current workflow, recalibration raised agreement with merge/abandon outcomes; on 16 older-workflow diffs, the previous judge performed better.","excerpt":"on 17 independently held-out diffs from the current workflow ($\\pi$ 0.16 to 0.51); on 16 diffs from older workflows, the previous judge stayed ahead","support":"source-stated"},{"text":"The authors say abstention can flag changes whose outcomes the judge's input cannot explain, indicating that better inputs may be needed rather than a better prompt.","excerpt":"flags changes whose outcome the judge's input cannot explain, telling a team when a judge needs better inputs rather than a better prompt.","support":"source-stated"}],"limitations":["The abstract reports evaluations on specific held-out datasets and diffs; it does not establish performance across other agents or workflows.","The abstract describes workflow outcome labels as noisy, so agreement with those labels does not by itself establish code quality."],"mediaId":"ea813f828db1bd3bc39bdd3e675a288e84501b558de29189f645ae9d66ff7c41","correctionReason":null,"id":"77a4861780d7ee5518dd5404aae6038795b5c7c7a0c550044db05e9fa1daad65","publicationId":"6d4158f5-cd06-47ce-82fd-f31f89a85e27","version":1,"sourceUrl":"https://arxiv.org/abs/2611.09291","sourceTitle":"AutoRefine: Auto-Calibrating Code Judges from Workflow Outcomes","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-06","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-09T05:03:49.666Z","createdAt":"2026-11-12T22:17:52.002Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"ea813f828db1bd3bc39bdd3e675a288e84501b558de29189f645ae9d66ff7c41","kind":"article","url":"https://arxiv.org/abs/2611.09291","sourceUrl":"https://arxiv.org/abs/2611.09291","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T22:16:45.142Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.09291","checkedAt":"2026-11-12T22:48:10.385Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"ea813f828db1bd3bc39bdd3e675a288e84501b558de29189f645ae9d66ff7c41","confidence":0.63,"provider":"typesafe","model":"jev-1.13.0","inputHash":"62ab0d84631d6e94e2e8075d69d272da45109e5a2c45b88a1f9c11d02fd1237c","decidedAt":"2026-11-12T22:16:46.208Z"},"provenance":{"textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observationId":"5edb2882cd0fc0791d74d4c016bf0725fec094fa560785e3277bdb6af70e07b3","retrievedAt":"2026-11-12T22:16:45.013Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"d67511cb-369b-4812-8372-136ed093a7bf"},"reviewer":{"model":"gpt-6-luna","runId":"6e67a227-6b63-4c2a-8983-7c2dbad7c95e"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-12T22:17:52.002Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Shahidi, Shervin","url":null,"kind":"unknown"},{"name":"Poon, James","url":null,"kind":"unknown"},{"name":"Xie, Hang","url":null,"kind":"unknown"},{"name":"Xu, Kenneth O.","url":null,"kind":"unknown"},{"name":"Håkanson, Robin","url":null,"kind":"unknown"},{"name":"Nair, Vikram","url":null,"kind":"unknown"},{"name":"Pillgram-Larsen, Jens","url":null,"kind":"unknown"},{"name":"Nagappan, Nachiappan","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Shahidi, Shervin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Shahidi, Shervin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09291","locator":"head/meta[name=author|citation_author]"},"assertionId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T22:48:10.385Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41","sourceUrl":"https://arxiv.org/abs/2611.09291","contentSha256":"7b3355425cb7e4036950d0a4305d46fb5dd9ccc882b81aacf350541bc09e73a1","textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observedAt":"2026-11-12T22:48:10.385Z"}},{"actor":{"name":"Poon, James","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Poon, James","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09291","locator":"head/meta[name=author|citation_author]"},"assertionId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T22:48:10.385Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41","sourceUrl":"https://arxiv.org/abs/2611.09291","contentSha256":"7b3355425cb7e4036950d0a4305d46fb5dd9ccc882b81aacf350541bc09e73a1","textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observedAt":"2026-11-12T22:48:10.385Z"}},{"actor":{"name":"Xie, Hang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Xie, Hang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09291","locator":"head/meta[name=author|citation_author]"},"assertionId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T22:48:10.385Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41","sourceUrl":"https://arxiv.org/abs/2611.09291","contentSha256":"7b3355425cb7e4036950d0a4305d46fb5dd9ccc882b81aacf350541bc09e73a1","textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observedAt":"2026-11-12T22:48:10.385Z"}},{"actor":{"name":"Xu, Kenneth O.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Xu, Kenneth O.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09291","locator":"head/meta[name=author|citation_author]"},"assertionId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T22:48:10.385Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41","sourceUrl":"https://arxiv.org/abs/2611.09291","contentSha256":"7b3355425cb7e4036950d0a4305d46fb5dd9ccc882b81aacf350541bc09e73a1","textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observedAt":"2026-11-12T22:48:10.385Z"}},{"actor":{"name":"Håkanson, Robin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Håkanson, Robin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09291","locator":"head/meta[name=author|citation_author]"},"assertionId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T22:48:10.385Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41","sourceUrl":"https://arxiv.org/abs/2611.09291","contentSha256":"7b3355425cb7e4036950d0a4305d46fb5dd9ccc882b81aacf350541bc09e73a1","textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observedAt":"2026-11-12T22:48:10.385Z"}},{"actor":{"name":"Nair, Vikram","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Nair, Vikram","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09291","locator":"head/meta[name=author|citation_author]"},"assertionId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T22:48:10.385Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41","sourceUrl":"https://arxiv.org/abs/2611.09291","contentSha256":"7b3355425cb7e4036950d0a4305d46fb5dd9ccc882b81aacf350541bc09e73a1","textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observedAt":"2026-11-12T22:48:10.385Z"}},{"actor":{"name":"Pillgram-Larsen, Jens","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Pillgram-Larsen, Jens","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09291","locator":"head/meta[name=author|citation_author]"},"assertionId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T22:48:10.385Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41","sourceUrl":"https://arxiv.org/abs/2611.09291","contentSha256":"7b3355425cb7e4036950d0a4305d46fb5dd9ccc882b81aacf350541bc09e73a1","textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observedAt":"2026-11-12T22:48:10.385Z"}},{"actor":{"name":"Nagappan, Nachiappan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Nagappan, Nachiappan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09291","locator":"head/meta[name=author|citation_author]"},"assertionId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T22:48:10.385Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41","sourceUrl":"https://arxiv.org/abs/2611.09291","contentSha256":"7b3355425cb7e4036950d0a4305d46fb5dd9ccc882b81aacf350541bc09e73a1","textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observedAt":"2026-11-12T22:48:10.385Z"}}],"participationCoverage":{"total":8,"returned":8,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.09291","snapshotId":"df3d7b4ddb407f10fa98f56d6d6afe0001a250d5aeef0798dc96103d92ee9e41","contentSha256":"7b3355425cb7e4036950d0a4305d46fb5dd9ccc882b81aacf350541bc09e73a1","textHash":"626931d77f1d32f5fe8508d024f6c8eb5f1eb526500b9945d411ff3ea68fe342","observedAt":"2026-11-12T22:48:10.385Z","basis":"publisher-page-metadata","sourceLinks":47,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/77a4861780d7ee5518dd5404aae6038795b5c7c7a0c550044db05e9fa1daad65/source"}}}],"audits":[{"publicationId":"6d4158f5-cd06-47ce-82fd-f31f89a85e27","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading consistently attributes the method, results and recalibration argument to the AutoRefine authors on arXiv. It does not present their claims as established fact or as Escape Meridian's view.","excerpt":"We present AutoRefine, a loop that rewrites a deployable judge's prompt to reproduce outcome labels"},{"dimension":"summary","status":"uncertain","explanation":"The core idea is mostly clear, but several central terms and details are not readable on their own. 'Teacher', 'diffs' and 'held-out' are unexplained. The summary never says the method rewrites the judge's prompt, yet later mentions 'a revised prompt'. It omits why judges matter here: some code changes have no test oracle. It also omits the noise result (25% to 20%). It says agreement improved 'with workflow outcomes', but the 0.36 to 0.60 result was measured on a dataset with an execution oracle. Smallest repairs: define the teacher, explain 'diffs', state that the prompt is rewritten, and separate the two evaluations.","excerpt":"LLM judges increasingly serve as quality gates for code changes that have no test oracle"},{"dimension":"date","status":"pass","explanation":"evidence.originalPublished is 2026-11-06, which matches the arXiv first-submission date for v1. The feed date of 2026-11-09 is correctly not used.","excerpt":"[Submitted on 6 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"Each claim matches a verbatim excerpt from the abstract. The older-workflow caveat is preserved. The question about performance as the agent evolves follows from the authors' own mixed result. The limitations avoid inventing benefits or harms. The caveat that agreement with labels does not establish code quality is a reasonable scoping note, grounded in the abstract's statement that the labels are noisy.","excerpt":"on 16 diffs from older workflows, the previous judge stayed ahead, illustrating the need for recalibration as the agent evolves"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified, direct arXiv abstract page for this paper, which is the most direct way to access the source.","excerpt":null}],"createdAt":"2026-11-12T22:17:52.002Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}