Luo, Yifan, Liu, Zhengying, Yuan, Yang, Yang, Zhilin, Yao, Andrew Chi-Chih · Nov 5, 2026
arXiv paper introduces TCSExam for evaluating LLMs across theoretical computer science
Markdown · JSON · Original source
Yifan Luo, Zhengying Liu, Yang Yuan, Zhilin Yang and Andrew Chi-Chih Yao introduce TCSExam, a intended to evaluate frontier across theoretical computer science research. It covers theorem proving, solver engineering and constructions, alongside direct attempts to resolve open conjectures. The authors say this approach captures both concrete research accomplishments and progress toward long-standing research goals. TCSExam-Full contains expert-reviewed tasks checked by the kernel or an executable task checker. The authors report that GPT-6 Astra leads Full and Opus 5.5 leads Lite, and that rankings across agents shared between the versions are strongly aligned. They also report improved solver outputs and mathematical constructions, along with independently verified special cases and bounds from open-conjecture attempts. The benchmark is designed to assess several kinds of research activity, not only theorem proving. Its results do not establish that LLMs can resolve open conjectures: the authors frame that as a question to study. They describe TCSExam as extensible and provide task sources, Docker build recipes and evaluation tools for reproducible use.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 5, 2026 (2026-11-05).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-08T20:08:41.471Z. First observed: 2026-11-06T05:10:29.390Z. Retrieved: 2026-11-08T20:04:14.291Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The abstract reports results, but does not establish that can resolve long-standing open conjectures; it presents that as a question the benchmark is meant to study.
Open question
As TCSExam is extended, will the rankings across its full and lite versions remain strongly aligned?
Source-stated
The authors introduce TCSExam to evaluate a broad range of theoretical computer science research, including several kinds of work and attempts on open conjectures.
It evaluates theorem proving, solver engineering, and constructions alongside direct attempts to resolve open conjectures
Source-stated
The full benchmark contains 100 expert-reviewed tasks, checked by the Lean kernel or an executable task checker.
TCSExam-Full contains 100 expert-reviewed tasks, with submissions checked by the Lean kernel or an executable task checker.
Source-stated
The authors report that GPT-6 Astra leads the full benchmark and Opus 5.5 leads the lite version.
GPT-6 Astra leads Full (71.9), and Opus 5.5 leads Lite (72.3).
Source-stated
The authors report strongly aligned rankings across agents appearing in both benchmark versions, supporting the lite benchmark as a compact, discriminative benchmark.
Full and Lite rankings are strongly aligned across eight shared agents (Spearman's $\rho=0.93$), supporting Lite as a compact, discriminative benchmark.
Source-stated
The evaluations yielded improved solver outputs and mathematical constructions, as well as independently verified special cases and bounds from open-conjecture attempts.
The evaluations yield improved solver outputs and mathematical constructions, together with independently verified special cases and bounds from open-conjecture attempts.
Source-stated
The authors describe TCSExam as extensible and provide materials for reproducible use.
TCSExam provides an extensible framework with task sources, Docker build recipes, and evaluation tools for reproducible use.
Source and contributor context
Luo, Yifan, Liu, Zhengying, Yuan, Yang, Yang, Zhilin, Yao, Andrew Chi-Chih · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Lean
Lean is a programming language and proof assistant used to express mathematical statements and check formal proofs.
Editorial provenance
Version 1. Publication ID: 5f068f51-bd4f-4cf5-818b-3e6c04060018.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-08T20:08:41.471Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: ad47667d8994a9b61f7381c09b4579de0b107162b356521250b44787cb716b9f. Source text hash: d554d63ec7fd09b99d7d36f9400f4b44cf6266d70a07f1f2c506ef7c31c34219.
Revision and source-check history
Version 1 · 2026-11-08T20:08:41.471Z
arXiv paper introduces TCSExam for evaluating LLMs across theoretical computer science
Yifan Luo, Zhengying Liu, Yang Yuan, Zhilin Yang and Andrew Chi-Chih Yao introduce TCSExam, a intended to evaluate frontier across theoretical computer science research. It covers theorem proving, solver engineering and constructions, alongside direct attempts to resolve open conjectures. The authors say this approach captures both concrete research accomplishments and progress toward long-standing research goals. TCSExam-Full contains expert-reviewed tasks checked by the kernel or an executable task checker. The authors report that GPT-6 Astra leads Full and Opus 5.5 leads Lite, and that rankings across agents shared between the versions are strongly aligned. They also report improved solver outputs and mathematical constructions, along with independently verified special cases and bounds from open-conjecture attempts. The benchmark is designed to assess several kinds of research activity, not only theorem proving. Its results do not establish that LLMs can resolve open conjectures: the authors frame that as a question to study. They describe TCSExam as extensible and provide task sources, Docker build recipes and evaluation tools for reproducible use.
Original source
cs.AI · Article
Source details
Original source date: Nov 5, 2026 (2026-11-05).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-08T20:08:41.471Z. First observed: 2026-11-06T05:10:29.390Z. Retrieved: 2026-11-08T20:04:14.291Z.
Limits
The abstract reports results, but does not establish that can resolve long-standing open conjectures; it presents that as a question the benchmark is meant to study.
Open question
As TCSExam is extended, will the rankings across its full and lite versions remain strongly aligned?
Source-stated
The authors introduce TCSExam to evaluate a broad range of theoretical computer science research, including several kinds of work and attempts on open conjectures.
It evaluates theorem proving, solver engineering, and constructions alongside direct attempts to resolve open conjectures
Source-stated
The full benchmark contains 100 expert-reviewed tasks, checked by the Lean kernel or an executable task checker.
TCSExam-Full contains 100 expert-reviewed tasks, with submissions checked by the Lean kernel or an executable task checker.
Source-stated
The authors report that GPT-6 Astra leads the full benchmark and Opus 5.5 leads the lite version.
GPT-6 Astra leads Full (71.9), and Opus 5.5 leads Lite (72.3).
Source-stated
The authors report strongly aligned rankings across agents appearing in both benchmark versions, supporting the lite benchmark as a compact, discriminative benchmark.
Full and Lite rankings are strongly aligned across eight shared agents (Spearman's $\rho=0.93$), supporting Lite as a compact, discriminative benchmark.
Source-stated
The evaluations yielded improved solver outputs and mathematical constructions, as well as independently verified special cases and bounds from open-conjecture attempts.
The evaluations yield improved solver outputs and mathematical constructions, together with independently verified special cases and bounds from open-conjecture attempts.
Source-stated
The authors describe TCSExam as extensible and provide materials for reproducible use.
TCSExam provides an extensible framework with task sources, Docker build recipes, and evaluation tools for reproducible use.
Source and contributor context
Luo, Yifan, Liu, Zhengying, Yuan, Yang, Yang, Zhilin, Yao, Andrew Chi-Chih · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Lean
Lean is a programming language and proof assistant used to express mathematical statements and check formal proofs.
Editorial provenance
Version 1. Publication ID: 5f068f51-bd4f-4cf5-818b-3e6c04060018.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-08T20:08:41.471Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: ad47667d8994a9b61f7381c09b4579de0b107162b356521250b44787cb716b9f. Source text hash: d554d63ec7fd09b99d7d36f9400f4b44cf6266d70a07f1f2c506ef7c31c34219.
uncertain · 2026-11-08T20:08:41.471Z · anthropic/claude-opus-5.5
Publication: 5f068f51-bd4f-4cf5-818b-3e6c04060018
attribution — pass: The reading names the five listed authors. It attributes the benchmark design, rankings, alignment result and research outputs to them with phrases such as 'the authors report'. It does not present these claims as established facts.
Authors:Yifan Luo, Zhengying Liu, Yang Yuan, Zhilin Yang, Andrew Chi-Chih Yao
summary — uncertain: The summary says 'Opus 5.5 leads Lite' and compares Full and Lite rankings, but never explains that Lite is a 20-task subset, five per track, meant to cut attempts by 80%. The question about Full and Lite alignment is therefore hard to follow. Central terms such as Lean kernel, solver engineering, constructions and open conjectures are also left unexplained for general readers. Smallest repair: define Lite in one clause and add tooltips or brief plain-language glosses for these terms.
TCSExam-Lite selects 20 tasks, five per track, reducing the number of task attempts by 80% at the same per-attempt budget.
date — pass: The displayed date, 2026-11-05, matches the arXiv v1 submission date. The feed entry is later and correctly is not used.
[Submitted on 5 Nov 2026]
support — pass: Each claim is quoted verbatim from the abstract. The limitation, that the source poses resolving conjectures as a question to study, matches the abstract's framing. The reading invents no benefits, costs or forecasts. The question about future alignment follows from the reported Spearman result without asserting an answer.
Can large language models (LLMs) advance theoretical computer science and resolve its long-standing open conjectures? To study progress toward this goal
media — pass: The selected media is the verified arXiv abstract page for this paper. It is the direct original source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceLean is a programming language and proof assistant used to express mathematical statements and check formal proofs.
Explore LeanarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceLean is a programming language and proof assistant used to express mathematical statements and check formal proofs.
Explore LeanarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition reference