# arXiv paper introduces TCSExam for evaluating LLMs across theoretical computer science [Canonical reading](https://escapemeridian.com/readings/arxiv-paper-introduces-tcsexam-for-evaluating-llms-across-cd901aee) Yifan Luo, Zhengying Liu, Yang Yuan, Zhilin Yang and Andrew Chi-Chih Yao introduce TCSExam, a benchmark intended to evaluate frontier LLMs across theoretical computer science research. It covers theorem proving, solver engineering and constructions, alongside direct attempts to resolve open conjectures. The authors say this approach captures both concrete research accomplishments and progress toward long-standing research goals. TCSExam-Full contains expert-reviewed tasks checked by the Lean kernel or an executable task checker. The authors report that GPT-6 Astra leads Full and Opus 5.5 leads Lite, and that rankings across agents shared between the versions are strongly aligned. They also report improved solver outputs and mathematical constructions, along with independently verified special cases and bounds from open-conjecture attempts. The benchmark is designed to assess several kinds of research activity, not only theorem proving. Its results do not establish that LLMs can resolve open conjectures: the authors frame that as a question to study. They describe TCSExam as extensible and provide task sources, Docker build recipes and evaluation tools for reproducible use. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Nov 5, 2026 (2026-11-05). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-08T20:08:41.471Z. First observed: 2026-11-06T05:10:29.390Z. Retrieved: 2026-11-08T20:04:14.291Z. - [TCSExam: Evaluating Frontier LLMs on the Full Range of TCS Research](https://arxiv.org/abs/2611.07962) ## Limits The abstract reports benchmark results, but does not establish that LLMs can resolve long-standing open conjectures; it presents that as a question the benchmark is meant to study. ## Open question As TCSExam is extended, will the rankings across its full and lite versions remain strongly aligned? ## Source-stated The authors introduce TCSExam to evaluate a broad range of theoretical computer science research, including several kinds of work and attempts on open conjectures. > It evaluates theorem proving, solver engineering, and constructions alongside direct attempts to resolve open conjectures - [Supporting source](https://arxiv.org/abs/2611.07962) ## Source-stated The full benchmark contains 100 expert-reviewed tasks, checked by the Lean kernel or an executable task checker. > TCSExam-Full contains 100 expert-reviewed tasks, with submissions checked by the Lean kernel or an executable task checker. - [Supporting source](https://arxiv.org/abs/2611.07962) ## Source-stated The authors report that GPT-6 Astra leads the full benchmark and Opus 5.5 leads the lite version. > GPT-6 Astra leads Full (71.9), and Opus 5.5 leads Lite (72.3). - [Supporting source](https://arxiv.org/abs/2611.07962) ## Source-stated The authors report strongly aligned rankings across agents appearing in both benchmark versions, supporting the lite benchmark as a compact, discriminative benchmark. > Full and Lite rankings are strongly aligned across eight shared agents (Spearman's $\\rho=0.93$), supporting Lite as a compact, discriminative benchmark. - [Supporting source](https://arxiv.org/abs/2611.07962) ## Source-stated The evaluations yielded improved solver outputs and mathematical constructions, as well as independently verified special cases and bounds from open-conjecture attempts. > The evaluations yield improved solver outputs and mathematical constructions, together with independently verified special cases and bounds from open-conjecture attempts. - [Supporting source](https://arxiv.org/abs/2611.07962) ## Source-stated The authors describe TCSExam as extensible and provide materials for reproducible use. > TCSExam provides an extensible framework with task sources, Docker build recipes, and evaluation tools for reproducible use. - [Supporting source](https://arxiv.org/abs/2611.07962) ## Source and contributor context Luo, Yifan, Liu, Zhengying, Yuan, Yang, Yang, Zhilin, Yao, Andrew Chi-Chih · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Lean Lean is a programming language and proof assistant used to express mathematical statements and check formal proofs. - [Explore this context](/entities/lean) - [Reference](https://lean-lang.org/) - [Reference](https://lean-lang.org/theorem_proving_in_lean4/Axioms-and-Computation/) ## Editorial provenance Version 1. Publication ID: 5f068f51-bd4f-4cf5-818b-3e6c04060018. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-08T20:08:41.471Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: ad47667d8994a9b61f7381c09b4579de0b107162b356521250b44787cb716b9f. Source text hash: d554d63ec7fd09b99d7d36f9400f4b44cf6266d70a07f1f2c506ef7c31c34219. ## Revision and source-check history ### Version 1 — 2026-11-08T20:08:41.471Z Yifan Luo, Zhengying Liu, Yang Yuan, Zhilin Yang and Andrew Chi-Chih Yao introduce TCSExam, a benchmark intended to evaluate frontier LLMs across theoretical computer science research. It covers theorem proving, solver engineering and constructions, alongside direct attempts to resolve open conjectures. The authors say this approach captures both concrete research accomplishments and progress toward long-standing research goals. TCSExam-Full contains expert-reviewed tasks checked by the Lean kernel or an executable task checker. The authors report that GPT-6 Astra leads Full and Opus 5.5 leads Lite, and that rankings across agents shared between the versions are strongly aligned. They also report improved solver outputs and mathematical constructions, along with independently verified special cases and bounds from open-conjecture attempts. The benchmark is designed to assess several kinds of research activity, not only theorem proving. Its results do not establish that LLMs can resolve open conjectures: the authors frame that as a question to study. They describe TCSExam as extensible and provide task sources, Docker build recipes and evaluation tools for reproducible use. ## Original source arXiv cs.AI · Article Original source date: Nov 5, 2026 (2026-11-05). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-08T20:08:41.471Z. First observed: 2026-11-06T05:10:29.390Z. Retrieved: 2026-11-08T20:04:14.291Z. - [TCSExam: Evaluating Frontier LLMs on the Full Range of TCS Research](https://arxiv.org/abs/2611.07962) ## Limits The abstract reports benchmark results, but does not establish that LLMs can resolve long-standing open conjectures; it presents that as a question the benchmark is meant to study. ## Open question As TCSExam is extended, will the rankings across its full and lite versions remain strongly aligned? ## Source-stated The authors introduce TCSExam to evaluate a broad range of theoretical computer science research, including several kinds of work and attempts on open conjectures. > It evaluates theorem proving, solver engineering, and constructions alongside direct attempts to resolve open conjectures - [Supporting source](https://arxiv.org/abs/2611.07962) ## Source-stated The full benchmark contains 100 expert-reviewed tasks, checked by the Lean kernel or an executable task checker. > TCSExam-Full contains 100 expert-reviewed tasks, with submissions checked by the Lean kernel or an executable task checker. - [Supporting source](https://arxiv.org/abs/2611.07962) ## Source-stated The authors report that GPT-6 Astra leads the full benchmark and Opus 5.5 leads the lite version. > GPT-6 Astra leads Full (71.9), and Opus 5.5 leads Lite (72.3). - [Supporting source](https://arxiv.org/abs/2611.07962) ## Source-stated The authors report strongly aligned rankings across agents appearing in both benchmark versions, supporting the lite benchmark as a compact, discriminative benchmark. > Full and Lite rankings are strongly aligned across eight shared agents (Spearman's $\\rho=0.93$), supporting Lite as a compact, discriminative benchmark. - [Supporting source](https://arxiv.org/abs/2611.07962) ## Source-stated The evaluations yielded improved solver outputs and mathematical constructions, as well as independently verified special cases and bounds from open-conjecture attempts. > The evaluations yield improved solver outputs and mathematical constructions, together with independently verified special cases and bounds from open-conjecture attempts. - [Supporting source](https://arxiv.org/abs/2611.07962) ## Source-stated The authors describe TCSExam as extensible and provide materials for reproducible use. > TCSExam provides an extensible framework with task sources, Docker build recipes, and evaluation tools for reproducible use. - [Supporting source](https://arxiv.org/abs/2611.07962) ## Source and contributor context Luo, Yifan, Liu, Zhengying, Yuan, Yang, Yang, Zhilin, Yao, Andrew Chi-Chih · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Lean Lean is a programming language and proof assistant used to express mathematical statements and check formal proofs. - [Explore this context](/entities/lean) - [Reference](https://lean-lang.org/) - [Reference](https://lean-lang.org/theorem_proving_in_lean4/Axioms-and-Computation/) ## Editorial provenance Version 1. Publication ID: 5f068f51-bd4f-4cf5-818b-3e6c04060018. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-08T20:08:41.471Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: ad47667d8994a9b61f7381c09b4579de0b107162b356521250b44787cb716b9f. Source text hash: d554d63ec7fd09b99d7d36f9400f4b44cf6266d70a07f1f2c506ef7c31c34219. ### uncertain — 2026-11-08T20:08:41.471Z — anthropic/claude-opus-5.5 Publication: 5f068f51-bd4f-4cf5-818b-3e6c04060018 attribution — pass: The reading names the five listed authors. It attributes the benchmark design, rankings, alignment result and research outputs to them with phrases such as 'the authors report'. It does not present these claims as established facts. > Authors:Yifan Luo, Zhengying Liu, Yang Yuan, Zhilin Yang, Andrew Chi-Chih Yao summary — uncertain: The summary says 'Opus 5.5 leads Lite' and compares Full and Lite rankings, but never explains that Lite is a 20-task subset, five per track, meant to cut attempts by 80%. The question about Full and Lite alignment is therefore hard to follow. Central terms such as Lean kernel, solver engineering, constructions and open conjectures are also left unexplained for general readers. Smallest repair: define Lite in one clause and add tooltips or brief plain-language glosses for these terms. > TCSExam-Lite selects 20 tasks, five per track, reducing the number of task attempts by 80% at the same per-attempt budget. date — pass: The displayed date, 2026-11-05, matches the arXiv v1 submission date. The feed entry is later and correctly is not used. > \[Submitted on 5 Nov 2026\] support — pass: Each claim is quoted verbatim from the abstract. The limitation, that the source poses resolving conjectures as a question to study, matches the abstract's framing. The reading invents no benefits, costs or forecasts. The question about future alignment follows from the reported Spearman result without asserting an answer. > Can large language models (LLMs) advance theoretical computer science and resolve its long-standing open conjectures? To study progress toward this goal media — pass: The selected media is the verified arXiv abstract page for this paper. It is the direct original source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/cd901aee338ff86575f1049fdc30f917a471e3f84d0cb60e784523005b00d7e1/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-paper-introduces-tcsexam-for-evaluating-llms-across-cd901aee.json)