Hamdi, Erfan, Shor, Joel, Lejeune, Emma · Nov 16, 2026
The authors find verification—not just code generation—limits scientific coding with LLMs
Markdown · JSON · Original source
Erfan Hamdi, Joel Shor and Emma Lejeune evaluate on 17 finite element tasks, including nonlinear and history-dependent mechanics and structural dynamics. Their central argument is that generating code is only part of the problem: models also need to produce results consistently and check them independently. In single-pass trials, the best model succeeds in 87.1% of implementation attempts, but the leading implementer and tester differ across tracks. Adding harnesses—which allow code execution and revision—helps on some tasks, but results vary by model and harness, and regressions occur. The contrast between Pass@3, at least one success in three attempts, and Pass$^3$, all three successes, highlights the consistency problem: Opus 5 with SimpleAgent reaches 96.5% Pass@3 but 77.1% Pass$^3$. The authors examine 249 failed agentic trajectories and identify inadequate test coverage and generic consistency checks as common verification failures; sometimes tests share an error with the implementation. A separate coder and tester configuration achieves the highest observed scores, but the authors describe it as exploratory. Their evidence points to independent verification and repeatable success as obstacles to trusting LLM-generated scientific code, not simply the ability to produce a working result once.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 16, 2026 (2026-11-16).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-20T05:34:54.460Z. First observed: 2026-11-20T05:04:35.067Z. Retrieved: 2026-11-20T05:34:02.840Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The authors evaluate a specific set of 17 finite element tasks; the abstract does not establish how results extend beyond those tasks. The separate coder and tester configuration is described as exploratory, so its reported scores do not establish that it will improve results consistently across models and tasks.
Open question
How consistently would a separate coder and tester improve results across models and finite element tasks?
Source-stated
The authors evaluate 15 models on 17 finite element tasks, including nonlinear and history-dependent mechanics and structural dynamics.
We extend FEM-Bench to 17 finite element tasks involving nonlinear and history-dependent mechanics and structural dynamics, and evaluate 15 models
Source-stated
The authors report that implementation and testing rankings differ, and that agentic execution improves some results but can also cause regressions.
rankings differ between tracks. The leading implementer achieves 49.9% joint test success, while the leading tester reaches 82.6%.
Source-stated
In their analysis of 249 failed agentic trajectories, inadequate test coverage and generic consistency checks are the most common self-verification failures.
The most common are inadequate test coverage (73.1%) and reliance on generic consistency checks (70.3%)
Source-stated
An exploratory configuration with separate coder and tester agents achieves the highest observed scores in the reported evaluation.
an exploratory configuration with separate coder and tester agents achieves the highest observed scores, with 97.6% Pass@3 and 88.8% Pass$^3$.
Source and contributor context
Hamdi, Erfan, Shor, Joel, Lejeune, Emma · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Agentic AI
AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: 0f5f8624-7ed2-421d-8a16-c7e99a2e6b4f.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-20T05:34:54.460Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: bb217d34410c614803e495dca1d64a1936221e128826535a260d0e1c0d31a5df. Source text hash: 2766ab91bb9171705a6565362d03a5d1dc6fa3687c8a1ddf67abdc399c24f022.
Revision and source-check history
Version 1 · 2026-11-20T05:34:54.460Z
The authors find verification—not just code generation—limits scientific coding with LLMs
Erfan Hamdi, Joel Shor and Emma Lejeune evaluate on 17 finite element tasks, including nonlinear and history-dependent mechanics and structural dynamics. Their central argument is that generating code is only part of the problem: models also need to produce results consistently and check them independently. In single-pass trials, the best model succeeds in 87.1% of implementation attempts, but the leading implementer and tester differ across tracks. Adding harnesses—which allow code execution and revision—helps on some tasks, but results vary by model and harness, and regressions occur. The contrast between Pass@3, at least one success in three attempts, and Pass$^3$, all three successes, highlights the consistency problem: Opus 5 with SimpleAgent reaches 96.5% Pass@3 but 77.1% Pass$^3$. The authors examine 249 failed agentic trajectories and identify inadequate test coverage and generic consistency checks as common verification failures; sometimes tests share an error with the implementation. A separate coder and tester configuration achieves the highest observed scores, but the authors describe it as exploratory. Their evidence points to independent verification and repeatable success as obstacles to trusting LLM-generated scientific code, not simply the ability to produce a working result once.
Original source
cs.AI · Article
Source details
Original source date: Nov 16, 2026 (2026-11-16).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-20T05:34:54.460Z. First observed: 2026-11-20T05:04:35.067Z. Retrieved: 2026-11-20T05:34:02.840Z.
Limits
The authors evaluate a specific set of 17 finite element tasks; the abstract does not establish how results extend beyond those tasks. The separate coder and tester configuration is described as exploratory, so its reported scores do not establish that it will improve results consistently across models and tasks.
Open question
How consistently would a separate coder and tester improve results across models and finite element tasks?
Source-stated
The authors evaluate 15 models on 17 finite element tasks, including nonlinear and history-dependent mechanics and structural dynamics.
We extend FEM-Bench to 17 finite element tasks involving nonlinear and history-dependent mechanics and structural dynamics, and evaluate 15 models
Source-stated
The authors report that implementation and testing rankings differ, and that agentic execution improves some results but can also cause regressions.
rankings differ between tracks. The leading implementer achieves 49.9% joint test success, while the leading tester reaches 82.6%.
Source-stated
In their analysis of 249 failed agentic trajectories, inadequate test coverage and generic consistency checks are the most common self-verification failures.
The most common are inadequate test coverage (73.1%) and reliance on generic consistency checks (70.3%)
Source-stated
An exploratory configuration with separate coder and tester agents achieves the highest observed scores in the reported evaluation.
an exploratory configuration with separate coder and tester agents achieves the highest observed scores, with 97.6% Pass@3 and 88.8% Pass$^3$.
Source and contributor context
Hamdi, Erfan, Shor, Joel, Lejeune, Emma · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Agentic AI
AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: 0f5f8624-7ed2-421d-8a16-c7e99a2e6b4f.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-20T05:34:54.460Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: bb217d34410c614803e495dca1d64a1936221e128826535a260d0e1c0d31a5df. Source text hash: 2766ab91bb9171705a6565362d03a5d1dc6fa3687c8a1ddf67abdc399c24f022.
passed · 2026-11-20T05:34:54.460Z · anthropic/claude-opus-5.5
Publication: 0f5f8624-7ed2-421d-8a16-c7e99a2e6b4f
attribution — pass: The reading names Erfan Hamdi, Joel Shor and Emma Lejeune as the authors, which matches the arXiv listing. It frames the results as the authors' findings and argument rather than as established fact.
Authors:Erfan Hamdi, Joel Shor, Emma Lejeune
summary — pass: The reading accurately conveys the paper's thesis: consistency across attempts and independent verification, not one-off generation, are the main obstacles. It explains agentic harnesses, Pass@3 and Pass^3 in plain language and labels the coder/tester setup as exploratory. The title stresses verification but drops consistency, though the body restores it. Smallest optional repair: note that single-pass trials ran without execution feedback. Finite element is a technical term that should be explained by a tooltip. The question follows from the source.
These results identify consistency across attempts and independent verification as the main obstacles to trusting LLM-generated scientific code.
date — pass: originalPublished is 2026-11-16, which matches the v1 submission date. The page shows a later v2 revision on 18 Nov and a feed date of 20 Nov, but the first submission date is the correct original publication date.
[Submitted on 16 Nov 2026 (v1), last revised 18 Nov 2026 (this version, v2)]
support — pass: Each figure is stated in the abstract: 87.1%, 96.5%/77.1%, the 249 trajectories, the 73.1% and 70.3% failure rates, and 97.6%/88.8%. So are the regressions and the shared errors between tests and implementations. Claim 2's excerpt covers only the ranking difference, but the regression point is stated elsewhere in the abstract. The limitations are appropriately cautious, and no unsupported benefits or impacts are implied.
Agentic execution solves some tasks that never succeeded in single-pass trials and improves consistency on others, but gains vary across models and harnesses, and regressions also occur.
media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the original source and the most direct way for readers to access it.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A language model with many learned numerical settings, used to process and generate language.
Definition referenceAI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referenceAI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv