{"id":"cd901aee338ff86575f1049fdc30f917a471e3f84d0cb60e784523005b00d7e1","title":"arXiv paper introduces TCSExam for evaluating LLMs across theoretical computer science","argument":"Yifan Luo, Zhengying Liu, Yang Yuan, Zhilin Yang and Andrew Chi-Chih Yao introduce TCSExam, a benchmark intended to evaluate frontier LLMs across theoretical computer science research. It covers theorem proving, solver engineering and constructions, alongside direct attempts to resolve open conjectures. The authors say this approach captures both concrete research accomplishments and progress toward long-standing research goals. TCSExam-Full contains expert-reviewed tasks checked by the Lean kernel or an executable task checker. The authors report that GPT-6 Astra leads Full and Opus 5.5 leads Lite, and that rankings across agents shared between the versions are strongly aligned. They also report improved solver outputs and mathematical constructions, along with independently verified special cases and bounds from open-conjecture attempts. The benchmark is designed to assess several kinds of research activity, not only theorem proving. Its results do not establish that LLMs can resolve open conjectures: the authors frame that as a question to study. They describe TCSExam as extensible and provide task sources, Docker build recipes and evaluation tools for reproducible use.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract reports benchmark results, but does not establish that LLMs can resolve long-standing open conjectures; it presents that as a question the benchmark is meant to study.","speaker":{"name":"Luo, Yifan, Liu, Zhengying, Yuan, Yang, Yang, Zhilin, Yao, Andrew Chi-Chih","roleAtSource":"Article author"},"source":{"title":"TCSExam: Evaluating Frontier LLMs on the Full Range of TCS Research","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.07962","publishedAt":"2026-11-05","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce TCSExam to evaluate a broad range of theoretical computer science research, including several kinds of work and attempts on open conjectures.","excerpt":"It evaluates theorem proving, solver engineering, and constructions alongside direct attempts to resolve open conjectures","url":"https://arxiv.org/abs/2611.07962","relationship":"Source-stated"},{"text":"The full benchmark contains 100 expert-reviewed tasks, checked by the Lean kernel or an executable task checker.","excerpt":"TCSExam-Full contains 100 expert-reviewed tasks, with submissions checked by the Lean kernel or an executable task checker.","url":"https://arxiv.org/abs/2611.07962","relationship":"Source-stated"},{"text":"The authors report that GPT-6 Astra leads the full benchmark and Opus 5.5 leads the lite version.","excerpt":"GPT-6 Astra leads Full (71.9), and Opus 5.5 leads Lite (72.3).","url":"https://arxiv.org/abs/2611.07962","relationship":"Source-stated"},{"text":"The authors report strongly aligned rankings across agents appearing in both benchmark versions, supporting the lite benchmark as a compact, discriminative benchmark.","excerpt":"Full and Lite rankings are strongly aligned across eight shared agents (Spearman's $\\rho=0.93$), supporting Lite as a compact, discriminative benchmark.","url":"https://arxiv.org/abs/2611.07962","relationship":"Source-stated"},{"text":"The evaluations yielded improved solver outputs and mathematical constructions, as well as independently verified special cases and bounds from open-conjecture attempts.","excerpt":"The evaluations yield improved solver outputs and mathematical constructions, together with independently verified special cases and bounds from open-conjecture attempts.","url":"https://arxiv.org/abs/2611.07962","relationship":"Source-stated"},{"text":"The authors describe TCSExam as extensible and provide materials for reproducible use.","excerpt":"TCSExam provides an extensible framework with task sources, Docker build recipes, and evaluation tools for reproducible use.","url":"https://arxiv.org/abs/2611.07962","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07962","checkedAt":"2026-11-08T20:40:11.540Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"As TCSExam is extended, will the rankings across its full and lite versions remain strongly aligned?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv paper introduces TCSExam for evaluating LLMs across theoretical computer science","summary":"Yifan Luo, Zhengying Liu, Yang Yuan, Zhilin Yang and Andrew Chi-Chih Yao introduce TCSExam, a benchmark intended to evaluate frontier LLMs across theoretical computer science research. It covers theorem proving, solver engineering and constructions, alongside direct attempts to resolve open conjectures. The authors say this approach captures both concrete research accomplishments and progress toward long-standing research goals. TCSExam-Full contains expert-reviewed tasks checked by the Lean kernel or an executable task checker. The authors report that GPT-6 Astra leads Full and Opus 5.5 leads Lite, and that rankings across agents shared between the versions are strongly aligned. They also report improved solver outputs and mathematical constructions, along with independently verified special cases and bounds from open-conjecture attempts. The benchmark is designed to assess several kinds of research activity, not only theorem proving. Its results do not establish that LLMs can resolve open conjectures: the authors frame that as a question to study. They describe TCSExam as extensible and provide task sources, Docker build recipes and evaluation tools for reproducible use.","question":"As TCSExam is extended, will the rankings across its full and lite versions remain strongly aligned?","claims":[{"text":"The authors introduce TCSExam to evaluate a broad range of theoretical computer science research, including several kinds of work and attempts on open conjectures.","excerpt":"It evaluates theorem proving, solver engineering, and constructions alongside direct attempts to resolve open conjectures","support":"source-stated"},{"text":"The full benchmark contains 100 expert-reviewed tasks, checked by the Lean kernel or an executable task checker.","excerpt":"TCSExam-Full contains 100 expert-reviewed tasks, with submissions checked by the Lean kernel or an executable task checker.","support":"source-stated"},{"text":"The authors report that GPT-6 Astra leads the full benchmark and Opus 5.5 leads the lite version.","excerpt":"GPT-6 Astra leads Full (71.9), and Opus 5.5 leads Lite (72.3).","support":"source-stated"},{"text":"The authors report strongly aligned rankings across agents appearing in both benchmark versions, supporting the lite benchmark as a compact, discriminative benchmark.","excerpt":"Full and Lite rankings are strongly aligned across eight shared agents (Spearman's $\\rho=0.93$), supporting Lite as a compact, discriminative benchmark.","support":"source-stated"},{"text":"The evaluations yielded improved solver outputs and mathematical constructions, as well as independently verified special cases and bounds from open-conjecture attempts.","excerpt":"The evaluations yield improved solver outputs and mathematical constructions, together with independently verified special cases and bounds from open-conjecture attempts.","support":"source-stated"},{"text":"The authors describe TCSExam as extensible and provide materials for reproducible use.","excerpt":"TCSExam provides an extensible framework with task sources, Docker build recipes, and evaluation tools for reproducible use.","support":"source-stated"}],"limitations":["The abstract reports benchmark results, but does not establish that LLMs can resolve long-standing open conjectures; it presents that as a question the benchmark is meant to study."],"mediaId":"337e2ee55490c12a81a465822a87276063b5857269a15a551db19a600b1f4a67","correctionReason":null,"id":"cd901aee338ff86575f1049fdc30f917a471e3f84d0cb60e784523005b00d7e1","publicationId":"5f068f51-bd4f-4cf5-818b-3e6c04060018","version":1,"sourceUrl":"https://arxiv.org/abs/2611.07962","sourceTitle":"TCSExam: Evaluating Frontier LLMs on the Full Range of TCS Research","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-05","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:10:29.390Z","createdAt":"2026-11-08T20:08:41.471Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"337e2ee55490c12a81a465822a87276063b5857269a15a551db19a600b1f4a67","kind":"article","url":"https://arxiv.org/abs/2611.07962","sourceUrl":"https://arxiv.org/abs/2611.07962","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-08T20:04:14.442Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07962","checkedAt":"2026-11-08T20:40:11.540Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"337e2ee55490c12a81a465822a87276063b5857269a15a551db19a600b1f4a67","confidence":0.9,"provider":"typesafe","model":"jev-1.13.0","inputHash":"037f5dfe14796bbbffe2605a87baeb8f39e1423749d50722dfa80ca88117cf23","decidedAt":"2026-11-08T20:04:15.345Z"},"provenance":{"textHash":"d554d63ec7fd09b99d7d36f9400f4b44cf6266d70a07f1f2c506ef7c31c34219","observationId":"ad47667d8994a9b61f7381c09b4579de0b107162b356521250b44787cb716b9f","retrievedAt":"2026-11-08T20:04:14.291Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"c726148d-b8d0-4dd6-813e-8dbc8562ccf2"},"reviewer":{"model":"gpt-6-luna","runId":"d263a2a8-aec8-45c1-86dd-25c050485e9a"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-08T20:08:41.471Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Luo, Yifan","url":null,"kind":"unknown"},{"name":"Liu, Zhengying","url":null,"kind":"unknown"},{"name":"Yuan, Yang","url":null,"kind":"unknown"},{"name":"Yang, Zhilin","url":null,"kind":"unknown"},{"name":"Yao, Andrew Chi-Chih","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Luo, Yifan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Luo, Yifan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07962","locator":"head/meta[name=author|citation_author]"},"assertionId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:40:11.540Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14","sourceUrl":"https://arxiv.org/abs/2611.07962","contentSha256":"682459f27afb755c9a0ae88c317648c238c9a67ef139c4b02fe6a987f08ad31c","textHash":"d554d63ec7fd09b99d7d36f9400f4b44cf6266d70a07f1f2c506ef7c31c34219","observedAt":"2026-11-08T20:40:11.540Z"}},{"actor":{"name":"Liu, Zhengying","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liu, Zhengying","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07962","locator":"head/meta[name=author|citation_author]"},"assertionId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:40:11.540Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14","sourceUrl":"https://arxiv.org/abs/2611.07962","contentSha256":"682459f27afb755c9a0ae88c317648c238c9a67ef139c4b02fe6a987f08ad31c","textHash":"d554d63ec7fd09b99d7d36f9400f4b44cf6266d70a07f1f2c506ef7c31c34219","observedAt":"2026-11-08T20:40:11.540Z"}},{"actor":{"name":"Yuan, Yang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yuan, Yang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07962","locator":"head/meta[name=author|citation_author]"},"assertionId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:40:11.540Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14","sourceUrl":"https://arxiv.org/abs/2611.07962","contentSha256":"682459f27afb755c9a0ae88c317648c238c9a67ef139c4b02fe6a987f08ad31c","textHash":"d554d63ec7fd09b99d7d36f9400f4b44cf6266d70a07f1f2c506ef7c31c34219","observedAt":"2026-11-08T20:40:11.540Z"}},{"actor":{"name":"Yang, Zhilin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yang, Zhilin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07962","locator":"head/meta[name=author|citation_author]"},"assertionId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:40:11.540Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14","sourceUrl":"https://arxiv.org/abs/2611.07962","contentSha256":"682459f27afb755c9a0ae88c317648c238c9a67ef139c4b02fe6a987f08ad31c","textHash":"d554d63ec7fd09b99d7d36f9400f4b44cf6266d70a07f1f2c506ef7c31c34219","observedAt":"2026-11-08T20:40:11.540Z"}},{"actor":{"name":"Yao, Andrew Chi-Chih","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yao, Andrew Chi-Chih","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07962","locator":"head/meta[name=author|citation_author]"},"assertionId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:40:11.540Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14","sourceUrl":"https://arxiv.org/abs/2611.07962","contentSha256":"682459f27afb755c9a0ae88c317648c238c9a67ef139c4b02fe6a987f08ad31c","textHash":"d554d63ec7fd09b99d7d36f9400f4b44cf6266d70a07f1f2c506ef7c31c34219","observedAt":"2026-11-08T20:40:11.540Z"}}],"participationCoverage":{"total":5,"returned":5,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.07962","snapshotId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14","contentSha256":"682459f27afb755c9a0ae88c317648c238c9a67ef139c4b02fe6a987f08ad31c","textHash":"d554d63ec7fd09b99d7d36f9400f4b44cf6266d70a07f1f2c506ef7c31c34219","observedAt":"2026-11-08T20:40:11.540Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/cd901aee338ff86575f1049fdc30f917a471e3f84d0cb60e784523005b00d7e1/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-paper-introduces-tcsexam-for-evaluating-llms-across-cd901aee","formats":{"html":"/readings/arxiv-paper-introduces-tcsexam-for-evaluating-llms-across-cd901aee","markdown":"/readings/arxiv-paper-introduces-tcsexam-for-evaluating-llms-across-cd901aee.md","json":"/readings/arxiv-paper-introduces-tcsexam-for-evaluating-llms-across-cd901aee.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"},{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"},{"id":"lean","name":"Lean","kind":"concept","context":"Lean is a programming language and proof assistant used to express mathematical statements and check formal proofs.","referenceUrls":["https://lean-lang.org/","https://lean-lang.org/theorem_proving_in_lean4/Axioms-and-Computation/"],"path":"/entities/lean","json":"/entities/lean.json","markdown":"/entities/lean.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"cd901aee338ff86575f1049fdc30f917a471e3f84d0cb60e784523005b00d7e1","revision":"5f068f51-bd4f-4cf5-818b-3e6c04060018","revisionKind":"publication-id","title":"arXiv paper introduces TCSExam for evaluating LLMs across theoretical computer science","url":"https://escapemeridian.com/readings/arxiv-paper-introduces-tcsexam-for-evaluating-llms-across-cd901aee"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.07962","revision":"d554d63ec7fd09b99d7d36f9400f4b44cf6266d70a07f1f2c506ef7c31c34219","revisionKind":"sha256-source-text","title":"TCSExam: Evaluating Frontier LLMs on the Full Range of TCS Research","url":"https://arxiv.org/abs/2611.07962"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"cd901aee338ff86575f1049fdc30f917a471e3f84d0cb60e784523005b00d7e1","title":"arXiv paper introduces TCSExam for evaluating LLMs across theoretical computer science","argument":"Yifan Luo, Zhengying Liu, Yang Yuan, Zhilin Yang and Andrew Chi-Chih Yao introduce TCSExam, a benchmark intended to evaluate frontier LLMs across theoretical computer science research. It covers theorem proving, solver engineering and constructions, alongside direct attempts to resolve open conjectures. The authors say this approach captures both concrete research accomplishments and progress toward long-standing research goals. TCSExam-Full contains expert-reviewed tasks checked by the Lean kernel or an executable task checker. The authors report that GPT-6 Astra leads Full and Opus 5.5 leads Lite, and that rankings across agents shared between the versions are strongly aligned. They also report improved solver outputs and mathematical constructions, along with independently verified special cases and bounds from open-conjecture attempts. The benchmark is designed to assess several kinds of research activity, not only theorem proving. Its results do not establish that LLMs can resolve open conjectures: the authors frame that as a question to study. They describe TCSExam as extensible and provide task sources, Docker build recipes and evaluation tools for reproducible use.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract reports benchmark results, but does not establish that LLMs can resolve long-standing open conjectures; it presents that as a question the benchmark is meant to study.","speaker":{"name":"Luo, Yifan, Liu, Zhengying, Yuan, Yang, Yang, Zhilin, Yao, Andrew Chi-Chih","roleAtSource":"Article author"},"source":{"title":"TCSExam: Evaluating Frontier LLMs on the Full Range of TCS Research","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.07962","publishedAt":"2026-11-05","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce TCSExam to evaluate a broad range of theoretical computer science research, including several kinds of work and attempts on open conjectures.","excerpt":"It evaluates theorem proving, solver engineering, and constructions alongside direct attempts to resolve open conjectures","url":"https://arxiv.org/abs/2611.07962","relationship":"Source-stated"},{"text":"The full benchmark contains 100 expert-reviewed tasks, checked by the Lean kernel or an executable task checker.","excerpt":"TCSExam-Full contains 100 expert-reviewed tasks, with submissions checked by the Lean kernel or an executable task checker.","url":"https://arxiv.org/abs/2611.07962","relationship":"Source-stated"},{"text":"The authors report that GPT-6 Astra leads the full benchmark and Opus 5.5 leads the lite version.","excerpt":"GPT-6 Astra leads Full (71.9), and Opus 5.5 leads Lite (72.3).","url":"https://arxiv.org/abs/2611.07962","relationship":"Source-stated"},{"text":"The authors report strongly aligned rankings across agents appearing in both benchmark versions, supporting the lite benchmark as a compact, discriminative benchmark.","excerpt":"Full and Lite rankings are strongly aligned across eight shared agents (Spearman's $\\rho=0.93$), supporting Lite as a compact, discriminative benchmark.","url":"https://arxiv.org/abs/2611.07962","relationship":"Source-stated"},{"text":"The evaluations yielded improved solver outputs and mathematical constructions, as well as independently verified special cases and bounds from open-conjecture attempts.","excerpt":"The evaluations yield improved solver outputs and mathematical constructions, together with independently verified special cases and bounds from open-conjecture attempts.","url":"https://arxiv.org/abs/2611.07962","relationship":"Source-stated"},{"text":"The authors describe TCSExam as extensible and provide materials for reproducible use.","excerpt":"TCSExam provides an extensible framework with task sources, Docker build recipes, and evaluation tools for reproducible use.","url":"https://arxiv.org/abs/2611.07962","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07962","checkedAt":"2026-11-08T20:40:11.540Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"As TCSExam is extended, will the rankings across its full and lite versions remain strongly aligned?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv paper introduces TCSExam for evaluating LLMs across theoretical computer science","summary":"Yifan Luo, Zhengying Liu, Yang Yuan, Zhilin Yang and Andrew Chi-Chih Yao introduce TCSExam, a benchmark intended to evaluate frontier LLMs across theoretical computer science research. It covers theorem proving, solver engineering and constructions, alongside direct attempts to resolve open conjectures. The authors say this approach captures both concrete research accomplishments and progress toward long-standing research goals. TCSExam-Full contains expert-reviewed tasks checked by the Lean kernel or an executable task checker. The authors report that GPT-6 Astra leads Full and Opus 5.5 leads Lite, and that rankings across agents shared between the versions are strongly aligned. They also report improved solver outputs and mathematical constructions, along with independently verified special cases and bounds from open-conjecture attempts. The benchmark is designed to assess several kinds of research activity, not only theorem proving. Its results do not establish that LLMs can resolve open conjectures: the authors frame that as a question to study. They describe TCSExam as extensible and provide task sources, Docker build recipes and evaluation tools for reproducible use.","question":"As TCSExam is extended, will the rankings across its full and lite versions remain strongly aligned?","claims":[{"text":"The authors introduce TCSExam to evaluate a broad range of theoretical computer science research, including several kinds of work and attempts on open conjectures.","excerpt":"It evaluates theorem proving, solver engineering, and constructions alongside direct attempts to resolve open conjectures","support":"source-stated"},{"text":"The full benchmark contains 100 expert-reviewed tasks, checked by the Lean kernel or an executable task checker.","excerpt":"TCSExam-Full contains 100 expert-reviewed tasks, with submissions checked by the Lean kernel or an executable task checker.","support":"source-stated"},{"text":"The authors report that GPT-6 Astra leads the full benchmark and Opus 5.5 leads the lite version.","excerpt":"GPT-6 Astra leads Full (71.9), and Opus 5.5 leads Lite (72.3).","support":"source-stated"},{"text":"The authors report strongly aligned rankings across agents appearing in both benchmark versions, supporting the lite benchmark as a compact, discriminative benchmark.","excerpt":"Full and Lite rankings are strongly aligned across eight shared agents (Spearman's $\\rho=0.93$), supporting Lite as a compact, discriminative benchmark.","support":"source-stated"},{"text":"The evaluations yielded improved solver outputs and mathematical constructions, as well as independently verified special cases and bounds from open-conjecture attempts.","excerpt":"The evaluations yield improved solver outputs and mathematical constructions, together with independently verified special cases and bounds from open-conjecture attempts.","support":"source-stated"},{"text":"The authors describe TCSExam as extensible and provide materials for reproducible use.","excerpt":"TCSExam provides an extensible framework with task sources, Docker build recipes, and evaluation tools for reproducible use.","support":"source-stated"}],"limitations":["The abstract reports benchmark results, but does not establish that LLMs can resolve long-standing open conjectures; it presents that as a question the benchmark is meant to study."],"mediaId":"337e2ee55490c12a81a465822a87276063b5857269a15a551db19a600b1f4a67","correctionReason":null,"id":"cd901aee338ff86575f1049fdc30f917a471e3f84d0cb60e784523005b00d7e1","publicationId":"5f068f51-bd4f-4cf5-818b-3e6c04060018","version":1,"sourceUrl":"https://arxiv.org/abs/2611.07962","sourceTitle":"TCSExam: Evaluating Frontier LLMs on the Full Range of TCS Research","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-05","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:10:29.390Z","createdAt":"2026-11-08T20:08:41.471Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"337e2ee55490c12a81a465822a87276063b5857269a15a551db19a600b1f4a67","kind":"article","url":"https://arxiv.org/abs/2611.07962","sourceUrl":"https://arxiv.org/abs/2611.07962","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-08T20:04:14.442Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07962","checkedAt":"2026-11-08T20:40:11.540Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"337e2ee55490c12a81a465822a87276063b5857269a15a551db19a600b1f4a67","confidence":0.9,"provider":"typesafe","model":"jev-1.13.0","inputHash":"037f5dfe14796bbbffe2605a87baeb8f39e1423749d50722dfa80ca88117cf23","decidedAt":"2026-11-08T20:04:15.345Z"},"provenance":{"textHash":"d554d63ec7fd09b99d7d36f9400f4b44cf6266d70a07f1f2c506ef7c31c34219","observationId":"ad47667d8994a9b61f7381c09b4579de0b107162b356521250b44787cb716b9f","retrievedAt":"2026-11-08T20:04:14.291Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"c726148d-b8d0-4dd6-813e-8dbc8562ccf2"},"reviewer":{"model":"gpt-6-luna","runId":"d263a2a8-aec8-45c1-86dd-25c050485e9a"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-08T20:08:41.471Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Luo, Yifan","url":null,"kind":"unknown"},{"name":"Liu, Zhengying","url":null,"kind":"unknown"},{"name":"Yuan, Yang","url":null,"kind":"unknown"},{"name":"Yang, Zhilin","url":null,"kind":"unknown"},{"name":"Yao, Andrew Chi-Chih","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Luo, Yifan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Luo, Yifan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07962","locator":"head/meta[name=author|citation_author]"},"assertionId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:40:11.540Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14","sourceUrl":"https://arxiv.org/abs/2611.07962","contentSha256":"682459f27afb755c9a0ae88c317648c238c9a67ef139c4b02fe6a987f08ad31c","textHash":"d554d63ec7fd09b99d7d36f9400f4b44cf6266d70a07f1f2c506ef7c31c34219","observedAt":"2026-11-08T20:40:11.540Z"}},{"actor":{"name":"Liu, Zhengying","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liu, Zhengying","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07962","locator":"head/meta[name=author|citation_author]"},"assertionId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:40:11.540Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14","sourceUrl":"https://arxiv.org/abs/2611.07962","contentSha256":"682459f27afb755c9a0ae88c317648c238c9a67ef139c4b02fe6a987f08ad31c","textHash":"d554d63ec7fd09b99d7d36f9400f4b44cf6266d70a07f1f2c506ef7c31c34219","observedAt":"2026-11-08T20:40:11.540Z"}},{"actor":{"name":"Yuan, Yang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yuan, Yang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07962","locator":"head/meta[name=author|citation_author]"},"assertionId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:40:11.540Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14","sourceUrl":"https://arxiv.org/abs/2611.07962","contentSha256":"682459f27afb755c9a0ae88c317648c238c9a67ef139c4b02fe6a987f08ad31c","textHash":"d554d63ec7fd09b99d7d36f9400f4b44cf6266d70a07f1f2c506ef7c31c34219","observedAt":"2026-11-08T20:40:11.540Z"}},{"actor":{"name":"Yang, Zhilin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yang, Zhilin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07962","locator":"head/meta[name=author|citation_author]"},"assertionId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:40:11.540Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14","sourceUrl":"https://arxiv.org/abs/2611.07962","contentSha256":"682459f27afb755c9a0ae88c317648c238c9a67ef139c4b02fe6a987f08ad31c","textHash":"d554d63ec7fd09b99d7d36f9400f4b44cf6266d70a07f1f2c506ef7c31c34219","observedAt":"2026-11-08T20:40:11.540Z"}},{"actor":{"name":"Yao, Andrew Chi-Chih","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yao, Andrew Chi-Chih","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07962","locator":"head/meta[name=author|citation_author]"},"assertionId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T20:40:11.540Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14","sourceUrl":"https://arxiv.org/abs/2611.07962","contentSha256":"682459f27afb755c9a0ae88c317648c238c9a67ef139c4b02fe6a987f08ad31c","textHash":"d554d63ec7fd09b99d7d36f9400f4b44cf6266d70a07f1f2c506ef7c31c34219","observedAt":"2026-11-08T20:40:11.540Z"}}],"participationCoverage":{"total":5,"returned":5,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.07962","snapshotId":"ac4688c248631099b38094db058fd66fc154759571c7b6c5d3ac6ffb16679c14","contentSha256":"682459f27afb755c9a0ae88c317648c238c9a67ef139c4b02fe6a987f08ad31c","textHash":"d554d63ec7fd09b99d7d36f9400f4b44cf6266d70a07f1f2c506ef7c31c34219","observedAt":"2026-11-08T20:40:11.540Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/cd901aee338ff86575f1049fdc30f917a471e3f84d0cb60e784523005b00d7e1/source"}}}],"audits":[{"publicationId":"5f068f51-bd4f-4cf5-818b-3e6c04060018","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading names the five listed authors. It attributes the benchmark design, rankings, alignment result and research outputs to them with phrases such as 'the authors report'. It does not present these claims as established facts.","excerpt":"Authors:Yifan Luo, Zhengying Liu, Yang Yuan, Zhilin Yang, Andrew Chi-Chih Yao"},{"dimension":"summary","status":"uncertain","explanation":"The summary says 'Opus 5.5 leads Lite' and compares Full and Lite rankings, but never explains that Lite is a 20-task subset, five per track, meant to cut attempts by 80%. The question about Full and Lite alignment is therefore hard to follow. Central terms such as Lean kernel, solver engineering, constructions and open conjectures are also left unexplained for general readers. Smallest repair: define Lite in one clause and add tooltips or brief plain-language glosses for these terms.","excerpt":"TCSExam-Lite selects 20 tasks, five per track, reducing the number of task attempts by 80% at the same per-attempt budget."},{"dimension":"date","status":"pass","explanation":"The displayed date, 2026-11-05, matches the arXiv v1 submission date. The feed entry is later and correctly is not used.","excerpt":"[Submitted on 5 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"Each claim is quoted verbatim from the abstract. The limitation, that the source poses resolving conjectures as a question to study, matches the abstract's framing. The reading invents no benefits, costs or forecasts. The question about future alignment follows from the reported Spearman result without asserting an answer.","excerpt":"Can large language models (LLMs) advance theoretical computer science and resolve its long-standing open conjectures? To study progress toward this goal"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified arXiv abstract page for this paper. It is the direct original source.","excerpt":null}],"createdAt":"2026-11-08T20:08:41.471Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}