{"id":"85ddc97b1375d50b59be0a09a48b0c50132f49bff42956736110f35ff0cdcf8c","title":"Can LLMs Reliably Grade Olympiad Proofs? A Study of Mathematical Verification with LLMs","argument":"The authors of this paper study whether large language models can grade Olympiad-level mathematical proofs, where each claim in a long chain of reasoning needs careful assessment. They introduce a human-curated dataset of USAMO 2025 candidate solutions with expert grading and evaluate existing LLM verifiers across frontier models. They also propose TROJAN, a process for generating adversarial proofs, and MABGrader, which treats grading as a choice among possible scores. The authors report that MABGrader outperforms ProofGrader by 26.4% in Quadratic Weighted Kappa and reduces grading error by 14%. As an Escape Meridian interpretation, these results suggest a possible way to improve automated grading, but the abstract does not establish how well the approach would work beyond the evaluated setting. How does MABGrader perform on proofs outside the USAMO 2025 dataset?","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The supplied evidence is the paper's abstract and does not provide details of the evaluation results or the dataset beyond its description. The abstract does not report how MABGrader performs on proofs outside the USAMO 2025 dataset.","speaker":{"name":"Ospanov, Azim, Feng, Zijin, Ding, Ding, Liu, Chengwu, Bai, Haoli, Sun, Jiacheng, Shang, Lifeng, Farnia, Farzan","roleAtSource":"Article author"},"source":{"title":"Can LLMs Reliably Grade Olympiad Proofs? A Study of Mathematical Verification with LLMs","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.36767","publishedAt":"2026-10-27","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors study LLM verification of Olympiad-level proofs, where long reasoning chains require assessment of each claim.","excerpt":"Olympiad-level proofs, which require meticulous assessment of each claim in a long reasoning chain.","url":"https://arxiv.org/abs/2610.36767","relationship":"Source-stated"},{"text":"The authors introduce a human-curated dataset of USAMO 2025 candidate solutions with expert grading.","excerpt":"a human-curated dataset of USAMO 2025 candidate solutions with expert grading","url":"https://arxiv.org/abs/2610.36767","relationship":"Source-stated"},{"text":"The authors propose TROJAN, an iterative self-critique pipeline for generating adversarial proofs at scale.","excerpt":"TROJAN, an iterative self-critique pipeline for generating high-fidelity adversarial proofs at scale.","url":"https://arxiv.org/abs/2610.36767","relationship":"Source-stated"},{"text":"The authors propose MABGrader, which frames grading as selecting an arm over a discrete score space.","excerpt":"MABGrader, a Multi-Armed Bandit framework that reframes grading as arm selection over the discrete score space.","url":"https://arxiv.org/abs/2610.36767","relationship":"Source-stated"},{"text":"The authors report that MABGrader outperforms ProofGrader by 26.4% in Quadratic Weighted Kappa and reduces grading error by 14%.","excerpt":"MABGrader outperforms the SOTA ProofGrader by 26.4% in Quadratic Weighted Kappa and reduces grading error by 14%.","url":"https://arxiv.org/abs/2610.36767","relationship":"Source-stated"},{"text":"The reported results suggest a possible route to improving automated proof grading, but do not establish performance outside the evaluated setting.","excerpt":"state-of-the-art models still leave considerable room for improvement. To narrow this gap, we propose MABGrader","url":"https://arxiv.org/abs/2610.36767","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.36767","checkedAt":"2026-11-05T02:10:06.541Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How does MABGrader perform on proofs outside the USAMO 2025 dataset?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Can LLMs Reliably Grade Olympiad Proofs? A Study of Mathematical Verification with LLMs","summary":"The authors of this paper study whether large language models can grade Olympiad-level mathematical proofs, where each claim in a long chain of reasoning needs careful assessment. They introduce a human-curated dataset of USAMO 2025 candidate solutions with expert grading and evaluate existing LLM verifiers across frontier models. They also propose TROJAN, a process for generating adversarial proofs, and MABGrader, which treats grading as a choice among possible scores. The authors report that MABGrader outperforms ProofGrader by 26.4% in Quadratic Weighted Kappa and reduces grading error by 14%. As an Escape Meridian interpretation, these results suggest a possible way to improve automated grading, but the abstract does not establish how well the approach would work beyond the evaluated setting. How does MABGrader perform on proofs outside the USAMO 2025 dataset?","question":"How does MABGrader perform on proofs outside the USAMO 2025 dataset?","claims":[{"text":"The authors study LLM verification of Olympiad-level proofs, where long reasoning chains require assessment of each claim.","excerpt":"Olympiad-level proofs, which require meticulous assessment of each claim in a long reasoning chain.","support":"source-stated"},{"text":"The authors introduce a human-curated dataset of USAMO 2025 candidate solutions with expert grading.","excerpt":"a human-curated dataset of USAMO 2025 candidate solutions with expert grading","support":"source-stated"},{"text":"The authors propose TROJAN, an iterative self-critique pipeline for generating adversarial proofs at scale.","excerpt":"TROJAN, an iterative self-critique pipeline for generating high-fidelity adversarial proofs at scale.","support":"source-stated"},{"text":"The authors propose MABGrader, which frames grading as selecting an arm over a discrete score space.","excerpt":"MABGrader, a Multi-Armed Bandit framework that reframes grading as arm selection over the discrete score space.","support":"source-stated"},{"text":"The authors report that MABGrader outperforms ProofGrader by 26.4% in Quadratic Weighted Kappa and reduces grading error by 14%.","excerpt":"MABGrader outperforms the SOTA ProofGrader by 26.4% in Quadratic Weighted Kappa and reduces grading error by 14%.","support":"source-stated"},{"text":"The reported results suggest a possible route to improving automated proof grading, but do not establish performance outside the evaluated setting.","excerpt":"state-of-the-art models still leave considerable room for improvement. To narrow this gap, we propose MABGrader","support":"interpretation"}],"limitations":["The supplied evidence is the paper's abstract and does not provide details of the evaluation results or the dataset beyond its description.","The abstract does not report how MABGrader performs on proofs outside the USAMO 2025 dataset."],"mediaId":"ca4a5eb96ecf444df6ea4200e512f8a0e51a24fc74e90faeb358d0cc433fbbcd","correctionReason":null,"id":"85ddc97b1375d50b59be0a09a48b0c50132f49bff42956736110f35ff0cdcf8c","publicationId":"2d0bffe0-c890-4e05-8375-2fba49fd71b1","version":1,"sourceUrl":"https://arxiv.org/abs/2610.36767","sourceTitle":"Can LLMs Reliably Grade Olympiad Proofs? A Study of Mathematical Verification with LLMs","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-27","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-28T05:52:45.748Z","createdAt":"2026-11-05T00:02:33.938Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"ca4a5eb96ecf444df6ea4200e512f8a0e51a24fc74e90faeb358d0cc433fbbcd","kind":"article","url":"https://arxiv.org/abs/2610.36767","sourceUrl":"https://arxiv.org/abs/2610.36767","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-30T03:31:37.990Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.36767","checkedAt":"2026-11-05T02:10:06.541Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"ca4a5eb96ecf444df6ea4200e512f8a0e51a24fc74e90faeb358d0cc433fbbcd","confidence":0.86,"provider":"typesafe","model":"jev-1.13.0","inputHash":"9395e170e2d785952149ab4f86e05070b2c89c221efad29a99d85a0bafea3aa9","decidedAt":"2026-10-30T03:31:38.982Z"},"provenance":{"textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observationId":"474735c627c126b075da04458a5ea1250d8b8921382f97c34cc6ef4e22483250","retrievedAt":"2026-10-30T03:31:37.835Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"76dc772d-4de1-4535-893a-6b58e6e4a4b0"},"reviewer":{"model":"gpt-6-luna","runId":"c6767409-b21d-4e52-84cd-04efc0157baa"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-05T00:02:33.938Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Ospanov, Azim","url":null,"kind":"unknown"},{"name":"Feng, Zijin","url":null,"kind":"unknown"},{"name":"Ding, Ding","url":null,"kind":"unknown"},{"name":"Liu, Chengwu","url":null,"kind":"unknown"},{"name":"Bai, Haoli","url":null,"kind":"unknown"},{"name":"Sun, Jiacheng","url":null,"kind":"unknown"},{"name":"Shang, Lifeng","url":null,"kind":"unknown"},{"name":"Farnia, Farzan","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Ospanov, Azim","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ospanov, Azim","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36767","locator":"head/meta[name=author|citation_author]"},"assertionId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:31:37.835Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1","sourceUrl":"https://arxiv.org/abs/2610.36767","contentSha256":"ead872e1eec1d3343dc29a88852aff10d3f94e5769a95bf31344632eb4a886f2","textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observedAt":"2026-10-30T03:31:37.835Z"}},{"actor":{"name":"Feng, Zijin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Feng, Zijin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36767","locator":"head/meta[name=author|citation_author]"},"assertionId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:31:37.835Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1","sourceUrl":"https://arxiv.org/abs/2610.36767","contentSha256":"ead872e1eec1d3343dc29a88852aff10d3f94e5769a95bf31344632eb4a886f2","textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observedAt":"2026-10-30T03:31:37.835Z"}},{"actor":{"name":"Ding, Ding","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ding, Ding","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36767","locator":"head/meta[name=author|citation_author]"},"assertionId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:31:37.835Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1","sourceUrl":"https://arxiv.org/abs/2610.36767","contentSha256":"ead872e1eec1d3343dc29a88852aff10d3f94e5769a95bf31344632eb4a886f2","textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observedAt":"2026-10-30T03:31:37.835Z"}},{"actor":{"name":"Liu, Chengwu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liu, Chengwu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36767","locator":"head/meta[name=author|citation_author]"},"assertionId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:31:37.835Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1","sourceUrl":"https://arxiv.org/abs/2610.36767","contentSha256":"ead872e1eec1d3343dc29a88852aff10d3f94e5769a95bf31344632eb4a886f2","textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observedAt":"2026-10-30T03:31:37.835Z"}},{"actor":{"name":"Bai, Haoli","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Bai, Haoli","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36767","locator":"head/meta[name=author|citation_author]"},"assertionId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:31:37.835Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1","sourceUrl":"https://arxiv.org/abs/2610.36767","contentSha256":"ead872e1eec1d3343dc29a88852aff10d3f94e5769a95bf31344632eb4a886f2","textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observedAt":"2026-10-30T03:31:37.835Z"}},{"actor":{"name":"Sun, Jiacheng","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sun, Jiacheng","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36767","locator":"head/meta[name=author|citation_author]"},"assertionId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:31:37.835Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1","sourceUrl":"https://arxiv.org/abs/2610.36767","contentSha256":"ead872e1eec1d3343dc29a88852aff10d3f94e5769a95bf31344632eb4a886f2","textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observedAt":"2026-10-30T03:31:37.835Z"}},{"actor":{"name":"Shang, Lifeng","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Shang, Lifeng","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36767","locator":"head/meta[name=author|citation_author]"},"assertionId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:31:37.835Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1","sourceUrl":"https://arxiv.org/abs/2610.36767","contentSha256":"ead872e1eec1d3343dc29a88852aff10d3f94e5769a95bf31344632eb4a886f2","textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observedAt":"2026-10-30T03:31:37.835Z"}},{"actor":{"name":"Farnia, Farzan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Farnia, Farzan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36767","locator":"head/meta[name=author|citation_author]"},"assertionId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:31:37.835Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1","sourceUrl":"https://arxiv.org/abs/2610.36767","contentSha256":"ead872e1eec1d3343dc29a88852aff10d3f94e5769a95bf31344632eb4a886f2","textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observedAt":"2026-10-30T03:31:37.835Z"}}],"participationCoverage":{"total":8,"returned":8,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.36767","snapshotId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1","contentSha256":"ead872e1eec1d3343dc29a88852aff10d3f94e5769a95bf31344632eb4a886f2","textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observedAt":"2026-10-30T03:31:37.835Z","basis":"publisher-page-metadata","sourceLinks":47,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/85ddc97b1375d50b59be0a09a48b0c50132f49bff42956736110f35ff0cdcf8c/source"}},"canonicalUrl":"https://escapemeridian.com/readings/can-llms-reliably-grade-olympiad-proofs-a-study-of-mathematical-85ddc97b","formats":{"html":"/readings/can-llms-reliably-grade-olympiad-proofs-a-study-of-mathematical-85ddc97b","markdown":"/readings/can-llms-reliably-grade-olympiad-proofs-a-study-of-mathematical-85ddc97b.md","json":"/readings/can-llms-reliably-grade-olympiad-proofs-a-study-of-mathematical-85ddc97b.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"},{"id":"term-frontier-model","name":"Frontier model","explanation":"One of the most capable AI models available at a given time; the set shifts as models improve.","referenceUrl":"https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"85ddc97b1375d50b59be0a09a48b0c50132f49bff42956736110f35ff0cdcf8c","revision":"2d0bffe0-c890-4e05-8375-2fba49fd71b1","revisionKind":"publication-id","title":"Can LLMs Reliably Grade Olympiad Proofs? A Study of Mathematical Verification with LLMs","url":"https://escapemeridian.com/readings/can-llms-reliably-grade-olympiad-proofs-a-study-of-mathematical-85ddc97b"},"source":{"type":"source","id":"https://arxiv.org/abs/2610.36767","revision":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","revisionKind":"sha256-source-text","title":"Can LLMs Reliably Grade Olympiad Proofs? A Study of Mathematical Verification with LLMs","url":"https://arxiv.org/abs/2610.36767"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"85ddc97b1375d50b59be0a09a48b0c50132f49bff42956736110f35ff0cdcf8c","title":"Can LLMs Reliably Grade Olympiad Proofs? A Study of Mathematical Verification with LLMs","argument":"The authors of this paper study whether large language models can grade Olympiad-level mathematical proofs, where each claim in a long chain of reasoning needs careful assessment. They introduce a human-curated dataset of USAMO 2025 candidate solutions with expert grading and evaluate existing LLM verifiers across frontier models. They also propose TROJAN, a process for generating adversarial proofs, and MABGrader, which treats grading as a choice among possible scores. The authors report that MABGrader outperforms ProofGrader by 26.4% in Quadratic Weighted Kappa and reduces grading error by 14%. As an Escape Meridian interpretation, these results suggest a possible way to improve automated grading, but the abstract does not establish how well the approach would work beyond the evaluated setting. How does MABGrader perform on proofs outside the USAMO 2025 dataset?","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The supplied evidence is the paper's abstract and does not provide details of the evaluation results or the dataset beyond its description. The abstract does not report how MABGrader performs on proofs outside the USAMO 2025 dataset.","speaker":{"name":"Ospanov, Azim, Feng, Zijin, Ding, Ding, Liu, Chengwu, Bai, Haoli, Sun, Jiacheng, Shang, Lifeng, Farnia, Farzan","roleAtSource":"Article author"},"source":{"title":"Can LLMs Reliably Grade Olympiad Proofs? A Study of Mathematical Verification with LLMs","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.36767","publishedAt":"2026-10-27","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors study LLM verification of Olympiad-level proofs, where long reasoning chains require assessment of each claim.","excerpt":"Olympiad-level proofs, which require meticulous assessment of each claim in a long reasoning chain.","url":"https://arxiv.org/abs/2610.36767","relationship":"Source-stated"},{"text":"The authors introduce a human-curated dataset of USAMO 2025 candidate solutions with expert grading.","excerpt":"a human-curated dataset of USAMO 2025 candidate solutions with expert grading","url":"https://arxiv.org/abs/2610.36767","relationship":"Source-stated"},{"text":"The authors propose TROJAN, an iterative self-critique pipeline for generating adversarial proofs at scale.","excerpt":"TROJAN, an iterative self-critique pipeline for generating high-fidelity adversarial proofs at scale.","url":"https://arxiv.org/abs/2610.36767","relationship":"Source-stated"},{"text":"The authors propose MABGrader, which frames grading as selecting an arm over a discrete score space.","excerpt":"MABGrader, a Multi-Armed Bandit framework that reframes grading as arm selection over the discrete score space.","url":"https://arxiv.org/abs/2610.36767","relationship":"Source-stated"},{"text":"The authors report that MABGrader outperforms ProofGrader by 26.4% in Quadratic Weighted Kappa and reduces grading error by 14%.","excerpt":"MABGrader outperforms the SOTA ProofGrader by 26.4% in Quadratic Weighted Kappa and reduces grading error by 14%.","url":"https://arxiv.org/abs/2610.36767","relationship":"Source-stated"},{"text":"The reported results suggest a possible route to improving automated proof grading, but do not establish performance outside the evaluated setting.","excerpt":"state-of-the-art models still leave considerable room for improvement. To narrow this gap, we propose MABGrader","url":"https://arxiv.org/abs/2610.36767","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.36767","checkedAt":"2026-11-05T02:10:06.541Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How does MABGrader perform on proofs outside the USAMO 2025 dataset?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Can LLMs Reliably Grade Olympiad Proofs? A Study of Mathematical Verification with LLMs","summary":"The authors of this paper study whether large language models can grade Olympiad-level mathematical proofs, where each claim in a long chain of reasoning needs careful assessment. They introduce a human-curated dataset of USAMO 2025 candidate solutions with expert grading and evaluate existing LLM verifiers across frontier models. They also propose TROJAN, a process for generating adversarial proofs, and MABGrader, which treats grading as a choice among possible scores. The authors report that MABGrader outperforms ProofGrader by 26.4% in Quadratic Weighted Kappa and reduces grading error by 14%. As an Escape Meridian interpretation, these results suggest a possible way to improve automated grading, but the abstract does not establish how well the approach would work beyond the evaluated setting. How does MABGrader perform on proofs outside the USAMO 2025 dataset?","question":"How does MABGrader perform on proofs outside the USAMO 2025 dataset?","claims":[{"text":"The authors study LLM verification of Olympiad-level proofs, where long reasoning chains require assessment of each claim.","excerpt":"Olympiad-level proofs, which require meticulous assessment of each claim in a long reasoning chain.","support":"source-stated"},{"text":"The authors introduce a human-curated dataset of USAMO 2025 candidate solutions with expert grading.","excerpt":"a human-curated dataset of USAMO 2025 candidate solutions with expert grading","support":"source-stated"},{"text":"The authors propose TROJAN, an iterative self-critique pipeline for generating adversarial proofs at scale.","excerpt":"TROJAN, an iterative self-critique pipeline for generating high-fidelity adversarial proofs at scale.","support":"source-stated"},{"text":"The authors propose MABGrader, which frames grading as selecting an arm over a discrete score space.","excerpt":"MABGrader, a Multi-Armed Bandit framework that reframes grading as arm selection over the discrete score space.","support":"source-stated"},{"text":"The authors report that MABGrader outperforms ProofGrader by 26.4% in Quadratic Weighted Kappa and reduces grading error by 14%.","excerpt":"MABGrader outperforms the SOTA ProofGrader by 26.4% in Quadratic Weighted Kappa and reduces grading error by 14%.","support":"source-stated"},{"text":"The reported results suggest a possible route to improving automated proof grading, but do not establish performance outside the evaluated setting.","excerpt":"state-of-the-art models still leave considerable room for improvement. To narrow this gap, we propose MABGrader","support":"interpretation"}],"limitations":["The supplied evidence is the paper's abstract and does not provide details of the evaluation results or the dataset beyond its description.","The abstract does not report how MABGrader performs on proofs outside the USAMO 2025 dataset."],"mediaId":"ca4a5eb96ecf444df6ea4200e512f8a0e51a24fc74e90faeb358d0cc433fbbcd","correctionReason":null,"id":"85ddc97b1375d50b59be0a09a48b0c50132f49bff42956736110f35ff0cdcf8c","publicationId":"2d0bffe0-c890-4e05-8375-2fba49fd71b1","version":1,"sourceUrl":"https://arxiv.org/abs/2610.36767","sourceTitle":"Can LLMs Reliably Grade Olympiad Proofs? A Study of Mathematical Verification with LLMs","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-27","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-28T05:52:45.748Z","createdAt":"2026-11-05T00:02:33.938Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"ca4a5eb96ecf444df6ea4200e512f8a0e51a24fc74e90faeb358d0cc433fbbcd","kind":"article","url":"https://arxiv.org/abs/2610.36767","sourceUrl":"https://arxiv.org/abs/2610.36767","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-30T03:31:37.990Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.36767","checkedAt":"2026-11-05T02:10:06.541Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"ca4a5eb96ecf444df6ea4200e512f8a0e51a24fc74e90faeb358d0cc433fbbcd","confidence":0.86,"provider":"typesafe","model":"jev-1.13.0","inputHash":"9395e170e2d785952149ab4f86e05070b2c89c221efad29a99d85a0bafea3aa9","decidedAt":"2026-10-30T03:31:38.982Z"},"provenance":{"textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observationId":"474735c627c126b075da04458a5ea1250d8b8921382f97c34cc6ef4e22483250","retrievedAt":"2026-10-30T03:31:37.835Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"76dc772d-4de1-4535-893a-6b58e6e4a4b0"},"reviewer":{"model":"gpt-6-luna","runId":"c6767409-b21d-4e52-84cd-04efc0157baa"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-05T00:02:33.938Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Ospanov, Azim","url":null,"kind":"unknown"},{"name":"Feng, Zijin","url":null,"kind":"unknown"},{"name":"Ding, Ding","url":null,"kind":"unknown"},{"name":"Liu, Chengwu","url":null,"kind":"unknown"},{"name":"Bai, Haoli","url":null,"kind":"unknown"},{"name":"Sun, Jiacheng","url":null,"kind":"unknown"},{"name":"Shang, Lifeng","url":null,"kind":"unknown"},{"name":"Farnia, Farzan","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Ospanov, Azim","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ospanov, Azim","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36767","locator":"head/meta[name=author|citation_author]"},"assertionId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:31:37.835Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1","sourceUrl":"https://arxiv.org/abs/2610.36767","contentSha256":"ead872e1eec1d3343dc29a88852aff10d3f94e5769a95bf31344632eb4a886f2","textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observedAt":"2026-10-30T03:31:37.835Z"}},{"actor":{"name":"Feng, Zijin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Feng, Zijin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36767","locator":"head/meta[name=author|citation_author]"},"assertionId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:31:37.835Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1","sourceUrl":"https://arxiv.org/abs/2610.36767","contentSha256":"ead872e1eec1d3343dc29a88852aff10d3f94e5769a95bf31344632eb4a886f2","textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observedAt":"2026-10-30T03:31:37.835Z"}},{"actor":{"name":"Ding, Ding","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ding, Ding","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36767","locator":"head/meta[name=author|citation_author]"},"assertionId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:31:37.835Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1","sourceUrl":"https://arxiv.org/abs/2610.36767","contentSha256":"ead872e1eec1d3343dc29a88852aff10d3f94e5769a95bf31344632eb4a886f2","textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observedAt":"2026-10-30T03:31:37.835Z"}},{"actor":{"name":"Liu, Chengwu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liu, Chengwu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36767","locator":"head/meta[name=author|citation_author]"},"assertionId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:31:37.835Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1","sourceUrl":"https://arxiv.org/abs/2610.36767","contentSha256":"ead872e1eec1d3343dc29a88852aff10d3f94e5769a95bf31344632eb4a886f2","textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observedAt":"2026-10-30T03:31:37.835Z"}},{"actor":{"name":"Bai, Haoli","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Bai, Haoli","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36767","locator":"head/meta[name=author|citation_author]"},"assertionId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:31:37.835Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1","sourceUrl":"https://arxiv.org/abs/2610.36767","contentSha256":"ead872e1eec1d3343dc29a88852aff10d3f94e5769a95bf31344632eb4a886f2","textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observedAt":"2026-10-30T03:31:37.835Z"}},{"actor":{"name":"Sun, Jiacheng","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sun, Jiacheng","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36767","locator":"head/meta[name=author|citation_author]"},"assertionId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:31:37.835Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1","sourceUrl":"https://arxiv.org/abs/2610.36767","contentSha256":"ead872e1eec1d3343dc29a88852aff10d3f94e5769a95bf31344632eb4a886f2","textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observedAt":"2026-10-30T03:31:37.835Z"}},{"actor":{"name":"Shang, Lifeng","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Shang, Lifeng","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36767","locator":"head/meta[name=author|citation_author]"},"assertionId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:31:37.835Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1","sourceUrl":"https://arxiv.org/abs/2610.36767","contentSha256":"ead872e1eec1d3343dc29a88852aff10d3f94e5769a95bf31344632eb4a886f2","textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observedAt":"2026-10-30T03:31:37.835Z"}},{"actor":{"name":"Farnia, Farzan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Farnia, Farzan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36767","locator":"head/meta[name=author|citation_author]"},"assertionId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:31:37.835Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1","sourceUrl":"https://arxiv.org/abs/2610.36767","contentSha256":"ead872e1eec1d3343dc29a88852aff10d3f94e5769a95bf31344632eb4a886f2","textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observedAt":"2026-10-30T03:31:37.835Z"}}],"participationCoverage":{"total":8,"returned":8,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.36767","snapshotId":"b881a357fbe3d65294eccbd01c9eec8e39f3988b07ff8ff0ca1bae2254b908b1","contentSha256":"ead872e1eec1d3343dc29a88852aff10d3f94e5769a95bf31344632eb4a886f2","textHash":"f6194b3b17d23802977b0311a80b92d12819df577fdf6c3b52862d3bd94cf658","observedAt":"2026-10-30T03:31:37.835Z","basis":"publisher-page-metadata","sourceLinks":47,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/85ddc97b1375d50b59be0a09a48b0c50132f49bff42956736110f35ff0cdcf8c/source"}}}],"audits":[{"publicationId":"2d0bffe0-c890-4e05-8375-2fba49fd71b1","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the dataset, the evaluation, TROJAN, MABGrader and the reported gains to the paper's authors. It labels the closing inference as Escape Meridian interpretation, consistent with the arXiv abstract.","excerpt":"Authors:Azim Ospanov, Zijin Feng, Ding Ding, Chengwu Liu, Haoli Bai, Jiacheng Sun, Lifeng Shang, Farzan Farnia"},{"dimension":"summary","status":"uncertain","explanation":"Several central terms are left unexplained for general readers: USAMO (a US national math olympiad), Quadratic Weighted Kappa (an agreement score), and adversarial proofs. Suggest term-library tooltips for these. The summary also swaps the exact term Multi-Armed Bandit for a loose paraphrase. It drops the SOTA label on ProofGrader. It omits two key points: the finding that current frontier models leave considerable room for improvement, and the stated motivation that expert inspection is costly to scale. Smallest repair: restore these points and the exact term. The question also repeats verbatim at the end of the summary.","excerpt":"Current proof verification thus primarily relies on expert inspection, which is costly to scale."},{"dimension":"date","status":"pass","explanation":"originalPublished is 2026-10-27, which matches the v1 submission date on the arXiv page. The 2026-10-29 feed date is correctly not used.","excerpt":"[Submitted on 27 Oct 2026]"},{"dimension":"support","status":"uncertain","explanation":"The source-stated claims match the abstract, and the interpretation is labeled and hedged. However, the question and limitations assume MABGrader's 26.4% and 14% gains were measured on the USAMO 2025 dataset. The abstract does not say which data the MABGrader comparison used. The phrase 'outside the USAMO 2025 dataset' could therefore misframe the evaluated setting. Smallest repair: say 'outside the evaluated setting'.","excerpt":"MABGrader outperforms the SOTA ProofGrader by 26.4% in Quadratic Weighted Kappa and reduces grading error by 14%."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the original arXiv abstract page. This is the most direct access to the source.","excerpt":null}],"createdAt":"2026-11-05T00:02:33.938Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}