Sorani, Alan, Narayanan, Ashvni, Shalyt, Michael, Monderer, Tali, Kalisch, Rotem, Barkan, Hila, Leibtag, Elyasheev, Weinbaum, Shachar, Schneider, Carsten, Campbell, John, Kaminer, Ido · Oct 29, 2026
Authors report results from an AI-assisted pipeline for evaluating multi-format proofs
Markdown · JSON · Original source
The authors report results from the Ramanujan Challenge, a of research-level number theory problems that includes unpublished results and open conjectures. They say submissions included correct solutions that differed conceptually from official solutions, as well as a proof of one published open conjecture. The paper also addresses how to judge complex proofs reliably at scale. Its AI-assisted pipeline evaluates natural language proofs, CAS code, and formalizations. Compared with one-shot judgment, the authors report fewer false-positive verdicts on natural language proofs and improved judgment accuracy on supporting code. The authors propose their approach as a methodology for evaluating research-level mathematics at scale, with potential applications across mathematics and AI for science. The abstract does not describe the evaluation set or provide enough detail to assess how the reported results might transfer to other problems.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 29, 2026 (2026-10-29).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-11T12:39:17.637Z. First observed: 2026-10-30T06:42:49.310Z. Retrieved: 2026-10-31T11:17:41.002Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The abstract does not describe the evaluation set or provide enough detail to assess how the reported results might transfer to other problems. The source gives headline comparison figures but does not specify in the abstract how the judgments were checked.
Open question
How well does the pipeline judge proofs beyond the Ramanujan Challenge submissions?
Source-stated
The authors describe the Ramanujan Challenge as a benchmark of research-level number theory problems, including unpublished results and open conjectures.
The Ramanujan Challenge is a benchmark comprised of research-level problems in number theory, including unpublished results and open conjectures.
Source-stated
The authors report correct submissions that differed conceptually from official solutions and a proof of one published open conjecture.
We find correct submissions that are conceptually distinct from the official solutions, and a proof of one of the published open conjectures.
Source-stated
The paper introduces an AI-assisted pipeline to assess natural language proofs, CAS code, and Lean formalizations.
We introduce an AI-assisted evaluation pipeline for natural language proofs, CAS code, and Lean formalizations that assesses semantic and logical correctness.
Source-stated
Compared with conventional one-shot LLM judgment, the authors report fewer false-positive verdicts on natural language proofs and better judgment accuracy on supporting code.
Compared with the conventional one-shot LLM judgment, our pipeline reduces false-positive verdicts on natural language proofs from 23% to 6% and improves judgment accuracy on supporting code from 46% to 58%.
Source and contributor context
Sorani, Alan, Narayanan, Ashvni, Shalyt, Michael, Monderer, Tali, Kalisch, Rotem, Barkan, Hila, Leibtag, Elyasheev, Weinbaum, Shachar, Schneider, Carsten, Campbell, John, Kaminer, Ido · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Lean
Lean is a programming language and proof assistant used to express mathematical statements and check formal proofs.
Editorial provenance
Version 1. Publication ID: 66ec6a68-cb00-4755-8819-c18133c94c45.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T12:39:17.637Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: c1146f0956499de93571a5e93d5670314a86b5d68b237cbddd00f9f104431833. Source text hash: cf9437a20e147baf64d67f19dd1fbfacce2c3eb8da550b63ff2a33312f618c02.
Revision and source-check history
Version 1 · 2026-11-11T12:39:17.637Z
Authors report results from an AI-assisted pipeline for evaluating multi-format proofs
The authors report results from the Ramanujan Challenge, a of research-level number theory problems that includes unpublished results and open conjectures. They say submissions included correct solutions that differed conceptually from official solutions, as well as a proof of one published open conjecture. The paper also addresses how to judge complex proofs reliably at scale. Its AI-assisted pipeline evaluates natural language proofs, CAS code, and formalizations. Compared with one-shot judgment, the authors report fewer false-positive verdicts on natural language proofs and improved judgment accuracy on supporting code. The authors propose their approach as a methodology for evaluating research-level mathematics at scale, with potential applications across mathematics and AI for science. The abstract does not describe the evaluation set or provide enough detail to assess how the reported results might transfer to other problems.
Original source
cs.AI · Article
Source details
Original source date: Oct 29, 2026 (2026-10-29).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-11T12:39:17.637Z. First observed: 2026-10-30T06:42:49.310Z. Retrieved: 2026-10-31T11:17:41.002Z.
Limits
The abstract does not describe the evaluation set or provide enough detail to assess how the reported results might transfer to other problems. The source gives headline comparison figures but does not specify in the abstract how the judgments were checked.
Open question
How well does the pipeline judge proofs beyond the Ramanujan Challenge submissions?
Source-stated
The authors describe the Ramanujan Challenge as a benchmark of research-level number theory problems, including unpublished results and open conjectures.
The Ramanujan Challenge is a benchmark comprised of research-level problems in number theory, including unpublished results and open conjectures.
Source-stated
The authors report correct submissions that differed conceptually from official solutions and a proof of one published open conjecture.
We find correct submissions that are conceptually distinct from the official solutions, and a proof of one of the published open conjectures.
Source-stated
The paper introduces an AI-assisted pipeline to assess natural language proofs, CAS code, and Lean formalizations.
We introduce an AI-assisted evaluation pipeline for natural language proofs, CAS code, and Lean formalizations that assesses semantic and logical correctness.
Source-stated
Compared with conventional one-shot LLM judgment, the authors report fewer false-positive verdicts on natural language proofs and better judgment accuracy on supporting code.
Compared with the conventional one-shot LLM judgment, our pipeline reduces false-positive verdicts on natural language proofs from 23% to 6% and improves judgment accuracy on supporting code from 46% to 58%.
Source and contributor context
Sorani, Alan, Narayanan, Ashvni, Shalyt, Michael, Monderer, Tali, Kalisch, Rotem, Barkan, Hila, Leibtag, Elyasheev, Weinbaum, Shachar, Schneider, Carsten, Campbell, John, Kaminer, Ido · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Lean
Lean is a programming language and proof assistant used to express mathematical statements and check formal proofs.
Editorial provenance
Version 1. Publication ID: 66ec6a68-cb00-4755-8819-c18133c94c45.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T12:39:17.637Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: c1146f0956499de93571a5e93d5670314a86b5d68b237cbddd00f9f104431833. Source text hash: cf9437a20e147baf64d67f19dd1fbfacce2c3eb8da550b63ff2a33312f618c02.
uncertain · 2026-11-11T12:39:17.637Z · anthropic/claude-opus-5.5
Publication: 66ec6a68-cb00-4755-8819-c18133c94c45
attribution — pass: The reading consistently attributes findings to the authors of the arXiv paper, using phrases such as 'the authors report' and 'they say'. It does not present the reported proof of an open conjecture or the pipeline gains as independently established facts.
Here, we analyze the Challenge submissions, measuring the relative success of different AI models, Computer Algebra Systems (CAS), and formal tools.
summary — uncertain: Central terms are left unexplained for a non-specialist: CAS (Computer Algebra Systems, never expanded), Lean formalizations, one-shot LLM judgment and false-positive verdicts. The summary also drops the actual figures (23% to 6%; 46% to 58%), which weakens the central result. It omits that the problems use formulas that can be checked numerically even when proving them is hard, and that the paper compares AI models, CAS and formal tools. Smallest repair: expand CAS, briefly gloss Lean and false positives, and restore the reported percentages.
It presented concrete formulas for mathematical constants whose correctness can be checked numerically, while their proofs may require non-obvious mathematics.
date — pass: originalPublished 2026-10-29 matches the v1 submission date on the arXiv page. The feed date of Oct 31 is a later announcement and is not used as the original date.
[Submitted on 29 Oct 2026]
support — pass: Each claim's excerpt matches the source text verbatim. The limitations accurately note that the abstract gives no evaluation-set details and no ground-truth method. The question about generalization follows from the evidence and is not a leading claim. The reading makes no unsupported benefit or cost claims, and the methodology is presented as the authors' proposal.
This approach provides a methodology for evaluating research-level mathematics at scale, with potential applications across mathematics and AI for science.
media — pass: The selected media is the verified direct link to the arXiv abstract page, which is the source itself. This is the most direct way for readers to access the paper.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceLean is a programming language and proof assistant used to express mathematical statements and check formal proofs.
Explore LeanA language model with many learned numerical settings, used to process and generate language.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceLean is a programming language and proof assistant used to express mathematical statements and check formal proofs.
Explore LeanA language model with many learned numerical settings, used to process and generate language.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv