{"id":"3f64330ddc5ccdac581903234c4b88468fc31d7aad99921d41ca6c3caea1eb4a","title":"The authors find verification—not just code generation—limits scientific coding with LLMs","argument":"Erfan Hamdi, Joel Shor and Emma Lejeune evaluate large language models on 17 finite element tasks, including nonlinear and history-dependent mechanics and structural dynamics. Their central argument is that generating code is only part of the problem: models also need to produce results consistently and check them independently. In single-pass trials, the best model succeeds in 87.1% of implementation attempts, but the leading implementer and tester differ across tracks. Adding agentic harnesses—which allow code execution and revision—helps on some tasks, but results vary by model and harness, and regressions occur. The contrast between Pass@3, at least one success in three attempts, and Pass$^3$, all three successes, highlights the consistency problem: Claude Opus 5 with SimpleAgent reaches 96.5% Pass@3 but 77.1% Pass$^3$. The authors examine 249 failed agentic trajectories and identify inadequate test coverage and generic consistency checks as common verification failures; sometimes tests share an error with the implementation. A separate coder and tester configuration achieves the highest observed scores, but the authors describe it as exploratory. Their evidence points to independent verification and repeatable success as obstacles to trusting LLM-generated scientific code, not simply the ability to produce a working result once.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The authors evaluate a specific set of 17 finite element tasks; the abstract does not establish how results extend beyond those tasks. The separate coder and tester configuration is described as exploratory, so its reported scores do not establish that it will improve results consistently across models and tasks.","speaker":{"name":"Hamdi, Erfan, Shor, Joel, Lejeune, Emma","roleAtSource":"Article author"},"source":{"title":"Scientific Coding with Large Language Models: Evidence from Finite Element Analysis","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.21176","publishedAt":"2026-11-16","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors evaluate 15 models on 17 finite element tasks, including nonlinear and history-dependent mechanics and structural dynamics.","excerpt":"We extend FEM-Bench to 17 finite element tasks involving nonlinear and history-dependent mechanics and structural dynamics, and evaluate 15 models","url":"https://arxiv.org/abs/2611.21176","relationship":"Source-stated"},{"text":"The authors report that implementation and testing rankings differ, and that agentic execution improves some results but can also cause regressions.","excerpt":"rankings differ between tracks. The leading implementer achieves 49.9% joint test success, while the leading tester reaches 82.6%.","url":"https://arxiv.org/abs/2611.21176","relationship":"Source-stated"},{"text":"In their analysis of 249 failed agentic trajectories, inadequate test coverage and generic consistency checks are the most common self-verification failures.","excerpt":"The most common are inadequate test coverage (73.1%) and reliance on generic consistency checks (70.3%)","url":"https://arxiv.org/abs/2611.21176","relationship":"Source-stated"},{"text":"An exploratory configuration with separate coder and tester agents achieves the highest observed scores in the reported evaluation.","excerpt":"an exploratory configuration with separate coder and tester agents achieves the highest observed scores, with 97.6% Pass@3 and 88.8% Pass$^3$.","url":"https://arxiv.org/abs/2611.21176","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.21176","checkedAt":"2026-11-20T22:38:09.785Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How consistently would a separate coder and tester improve results across models and finite element tasks?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"The authors find verification—not just code generation—limits scientific coding with LLMs","summary":"Erfan Hamdi, Joel Shor and Emma Lejeune evaluate large language models on 17 finite element tasks, including nonlinear and history-dependent mechanics and structural dynamics. Their central argument is that generating code is only part of the problem: models also need to produce results consistently and check them independently. In single-pass trials, the best model succeeds in 87.1% of implementation attempts, but the leading implementer and tester differ across tracks. Adding agentic harnesses—which allow code execution and revision—helps on some tasks, but results vary by model and harness, and regressions occur. The contrast between Pass@3, at least one success in three attempts, and Pass$^3$, all three successes, highlights the consistency problem: Claude Opus 5 with SimpleAgent reaches 96.5% Pass@3 but 77.1% Pass$^3$. The authors examine 249 failed agentic trajectories and identify inadequate test coverage and generic consistency checks as common verification failures; sometimes tests share an error with the implementation. A separate coder and tester configuration achieves the highest observed scores, but the authors describe it as exploratory. Their evidence points to independent verification and repeatable success as obstacles to trusting LLM-generated scientific code, not simply the ability to produce a working result once.","question":"How consistently would a separate coder and tester improve results across models and finite element tasks?","claims":[{"text":"The authors evaluate 15 models on 17 finite element tasks, including nonlinear and history-dependent mechanics and structural dynamics.","excerpt":"We extend FEM-Bench to 17 finite element tasks involving nonlinear and history-dependent mechanics and structural dynamics, and evaluate 15 models","support":"source-stated"},{"text":"The authors report that implementation and testing rankings differ, and that agentic execution improves some results but can also cause regressions.","excerpt":"rankings differ between tracks. The leading implementer achieves 49.9% joint test success, while the leading tester reaches 82.6%.","support":"source-stated"},{"text":"In their analysis of 249 failed agentic trajectories, inadequate test coverage and generic consistency checks are the most common self-verification failures.","excerpt":"The most common are inadequate test coverage (73.1%) and reliance on generic consistency checks (70.3%)","support":"source-stated"},{"text":"An exploratory configuration with separate coder and tester agents achieves the highest observed scores in the reported evaluation.","excerpt":"an exploratory configuration with separate coder and tester agents achieves the highest observed scores, with 97.6% Pass@3 and 88.8% Pass$^3$.","support":"source-stated"}],"limitations":["The authors evaluate a specific set of 17 finite element tasks; the abstract does not establish how results extend beyond those tasks.","The separate coder and tester configuration is described as exploratory, so its reported scores do not establish that it will improve results consistently across models and tasks."],"mediaId":"241a6a3a650e0124353c723a0ebb84934449a4839d362fc98686871ec20a4445","correctionReason":null,"id":"3f64330ddc5ccdac581903234c4b88468fc31d7aad99921d41ca6c3caea1eb4a","publicationId":"0f5f8624-7ed2-421d-8a16-c7e99a2e6b4f","version":1,"sourceUrl":"https://arxiv.org/abs/2611.21176","sourceTitle":"Scientific Coding with Large Language Models: Evidence from Finite Element Analysis","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-16","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-20T05:04:35.067Z","createdAt":"2026-11-20T05:34:54.460Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"241a6a3a650e0124353c723a0ebb84934449a4839d362fc98686871ec20a4445","kind":"article","url":"https://arxiv.org/abs/2611.21176","sourceUrl":"https://arxiv.org/abs/2611.21176","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-20T05:34:03.114Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.21176","checkedAt":"2026-11-20T22:38:09.785Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"241a6a3a650e0124353c723a0ebb84934449a4839d362fc98686871ec20a4445","confidence":0.64,"provider":"typesafe","model":"jev-1.13.0","inputHash":"206c298b8237213adec7b1dce4430c28223d8e47c146e951471f7168f9928c5f","decidedAt":"2026-11-20T05:34:05.068Z"},"provenance":{"textHash":"2766ab91bb9171705a6565362d03a5d1dc6fa3687c8a1ddf67abdc399c24f022","observationId":"bb217d34410c614803e495dca1d64a1936221e128826535a260d0e1c0d31a5df","retrievedAt":"2026-11-20T05:34:02.840Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"deaa029a-3bd2-4479-885d-d0d045a8065f"},"reviewer":{"model":"gpt-6-luna","runId":"b038498b-7b5f-4b15-839f-6abf69aca7e3"}},"audit":{"status":"passed","reviewedAt":"2026-11-20T05:34:54.460Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Hamdi, Erfan","url":null,"kind":"unknown"},{"name":"Shor, Joel","url":null,"kind":"unknown"},{"name":"Lejeune, Emma","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Hamdi, Erfan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Hamdi, Erfan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21176","locator":"head/meta[name=author|citation_author]"},"assertionId":"eb322beaba26500bc6915d1ac2d37d1a5ff509fe72b3a38879c4e227c99475f7:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-20T22:38:09.785Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"eb322beaba26500bc6915d1ac2d37d1a5ff509fe72b3a38879c4e227c99475f7","sourceUrl":"https://arxiv.org/abs/2611.21176","contentSha256":"5534b3e1a7af23eb47d275bb3bf9484f1c7f89d5e9d780bc4c37ec9cf56fe60b","textHash":"2766ab91bb9171705a6565362d03a5d1dc6fa3687c8a1ddf67abdc399c24f022","observedAt":"2026-11-20T22:38:09.785Z"}},{"actor":{"name":"Shor, Joel","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Shor, Joel","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21176","locator":"head/meta[name=author|citation_author]"},"assertionId":"eb322beaba26500bc6915d1ac2d37d1a5ff509fe72b3a38879c4e227c99475f7:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-20T22:38:09.785Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"eb322beaba26500bc6915d1ac2d37d1a5ff509fe72b3a38879c4e227c99475f7","sourceUrl":"https://arxiv.org/abs/2611.21176","contentSha256":"5534b3e1a7af23eb47d275bb3bf9484f1c7f89d5e9d780bc4c37ec9cf56fe60b","textHash":"2766ab91bb9171705a6565362d03a5d1dc6fa3687c8a1ddf67abdc399c24f022","observedAt":"2026-11-20T22:38:09.785Z"}},{"actor":{"name":"Lejeune, Emma","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Lejeune, Emma","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21176","locator":"head/meta[name=author|citation_author]"},"assertionId":"eb322beaba26500bc6915d1ac2d37d1a5ff509fe72b3a38879c4e227c99475f7:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-20T22:38:09.785Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"eb322beaba26500bc6915d1ac2d37d1a5ff509fe72b3a38879c4e227c99475f7","sourceUrl":"https://arxiv.org/abs/2611.21176","contentSha256":"5534b3e1a7af23eb47d275bb3bf9484f1c7f89d5e9d780bc4c37ec9cf56fe60b","textHash":"2766ab91bb9171705a6565362d03a5d1dc6fa3687c8a1ddf67abdc399c24f022","observedAt":"2026-11-20T22:38:09.785Z"}}],"participationCoverage":{"total":3,"returned":3,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.21176","snapshotId":"eb322beaba26500bc6915d1ac2d37d1a5ff509fe72b3a38879c4e227c99475f7","contentSha256":"5534b3e1a7af23eb47d275bb3bf9484f1c7f89d5e9d780bc4c37ec9cf56fe60b","textHash":"2766ab91bb9171705a6565362d03a5d1dc6fa3687c8a1ddf67abdc399c24f022","observedAt":"2026-11-20T22:38:09.785Z","basis":"publisher-page-metadata","sourceLinks":45,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/3f64330ddc5ccdac581903234c4b88468fc31d7aad99921d41ca6c3caea1eb4a/source"}},"canonicalUrl":"https://escapemeridian.com/readings/the-authors-find-verification-not-just-code-generation-limits-3f64330d","formats":{"html":"/readings/the-authors-find-verification-not-just-code-generation-limits-3f64330d","markdown":"/readings/the-authors-find-verification-not-just-code-generation-limits-3f64330d.md","json":"/readings/the-authors-find-verification-not-just-code-generation-limits-3f64330d.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"},{"id":"term-agentic","name":"Agentic AI","explanation":"AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.","referenceUrl":"https://developers.google.com/machine-learning/glossary#agentic"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"},{"id":"claude","name":"Claude","kind":"model","context":"Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.","referenceUrls":["https://claude.com/product/overview","https://www.anthropic.com/news","https://www.anthropic.com/research/team/economics"],"path":"/entities/claude","json":"/entities/claude.json","markdown":"/entities/claude.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"3f64330ddc5ccdac581903234c4b88468fc31d7aad99921d41ca6c3caea1eb4a","revision":"0f5f8624-7ed2-421d-8a16-c7e99a2e6b4f","revisionKind":"publication-id","title":"The authors find verification—not just code generation—limits scientific coding with LLMs","url":"https://escapemeridian.com/readings/the-authors-find-verification-not-just-code-generation-limits-3f64330d"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.21176","revision":"2766ab91bb9171705a6565362d03a5d1dc6fa3687c8a1ddf67abdc399c24f022","revisionKind":"sha256-source-text","title":"Scientific Coding with Large Language Models: Evidence from Finite Element Analysis","url":"https://arxiv.org/abs/2611.21176"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"3f64330ddc5ccdac581903234c4b88468fc31d7aad99921d41ca6c3caea1eb4a","title":"The authors find verification—not just code generation—limits scientific coding with LLMs","argument":"Erfan Hamdi, Joel Shor and Emma Lejeune evaluate large language models on 17 finite element tasks, including nonlinear and history-dependent mechanics and structural dynamics. Their central argument is that generating code is only part of the problem: models also need to produce results consistently and check them independently. In single-pass trials, the best model succeeds in 87.1% of implementation attempts, but the leading implementer and tester differ across tracks. Adding agentic harnesses—which allow code execution and revision—helps on some tasks, but results vary by model and harness, and regressions occur. The contrast between Pass@3, at least one success in three attempts, and Pass$^3$, all three successes, highlights the consistency problem: Claude Opus 5 with SimpleAgent reaches 96.5% Pass@3 but 77.1% Pass$^3$. The authors examine 249 failed agentic trajectories and identify inadequate test coverage and generic consistency checks as common verification failures; sometimes tests share an error with the implementation. A separate coder and tester configuration achieves the highest observed scores, but the authors describe it as exploratory. Their evidence points to independent verification and repeatable success as obstacles to trusting LLM-generated scientific code, not simply the ability to produce a working result once.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The authors evaluate a specific set of 17 finite element tasks; the abstract does not establish how results extend beyond those tasks. The separate coder and tester configuration is described as exploratory, so its reported scores do not establish that it will improve results consistently across models and tasks.","speaker":{"name":"Hamdi, Erfan, Shor, Joel, Lejeune, Emma","roleAtSource":"Article author"},"source":{"title":"Scientific Coding with Large Language Models: Evidence from Finite Element Analysis","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.21176","publishedAt":"2026-11-16","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors evaluate 15 models on 17 finite element tasks, including nonlinear and history-dependent mechanics and structural dynamics.","excerpt":"We extend FEM-Bench to 17 finite element tasks involving nonlinear and history-dependent mechanics and structural dynamics, and evaluate 15 models","url":"https://arxiv.org/abs/2611.21176","relationship":"Source-stated"},{"text":"The authors report that implementation and testing rankings differ, and that agentic execution improves some results but can also cause regressions.","excerpt":"rankings differ between tracks. The leading implementer achieves 49.9% joint test success, while the leading tester reaches 82.6%.","url":"https://arxiv.org/abs/2611.21176","relationship":"Source-stated"},{"text":"In their analysis of 249 failed agentic trajectories, inadequate test coverage and generic consistency checks are the most common self-verification failures.","excerpt":"The most common are inadequate test coverage (73.1%) and reliance on generic consistency checks (70.3%)","url":"https://arxiv.org/abs/2611.21176","relationship":"Source-stated"},{"text":"An exploratory configuration with separate coder and tester agents achieves the highest observed scores in the reported evaluation.","excerpt":"an exploratory configuration with separate coder and tester agents achieves the highest observed scores, with 97.6% Pass@3 and 88.8% Pass$^3$.","url":"https://arxiv.org/abs/2611.21176","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.21176","checkedAt":"2026-11-20T22:38:09.785Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How consistently would a separate coder and tester improve results across models and finite element tasks?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"The authors find verification—not just code generation—limits scientific coding with LLMs","summary":"Erfan Hamdi, Joel Shor and Emma Lejeune evaluate large language models on 17 finite element tasks, including nonlinear and history-dependent mechanics and structural dynamics. Their central argument is that generating code is only part of the problem: models also need to produce results consistently and check them independently. In single-pass trials, the best model succeeds in 87.1% of implementation attempts, but the leading implementer and tester differ across tracks. Adding agentic harnesses—which allow code execution and revision—helps on some tasks, but results vary by model and harness, and regressions occur. The contrast between Pass@3, at least one success in three attempts, and Pass$^3$, all three successes, highlights the consistency problem: Claude Opus 5 with SimpleAgent reaches 96.5% Pass@3 but 77.1% Pass$^3$. The authors examine 249 failed agentic trajectories and identify inadequate test coverage and generic consistency checks as common verification failures; sometimes tests share an error with the implementation. A separate coder and tester configuration achieves the highest observed scores, but the authors describe it as exploratory. Their evidence points to independent verification and repeatable success as obstacles to trusting LLM-generated scientific code, not simply the ability to produce a working result once.","question":"How consistently would a separate coder and tester improve results across models and finite element tasks?","claims":[{"text":"The authors evaluate 15 models on 17 finite element tasks, including nonlinear and history-dependent mechanics and structural dynamics.","excerpt":"We extend FEM-Bench to 17 finite element tasks involving nonlinear and history-dependent mechanics and structural dynamics, and evaluate 15 models","support":"source-stated"},{"text":"The authors report that implementation and testing rankings differ, and that agentic execution improves some results but can also cause regressions.","excerpt":"rankings differ between tracks. The leading implementer achieves 49.9% joint test success, while the leading tester reaches 82.6%.","support":"source-stated"},{"text":"In their analysis of 249 failed agentic trajectories, inadequate test coverage and generic consistency checks are the most common self-verification failures.","excerpt":"The most common are inadequate test coverage (73.1%) and reliance on generic consistency checks (70.3%)","support":"source-stated"},{"text":"An exploratory configuration with separate coder and tester agents achieves the highest observed scores in the reported evaluation.","excerpt":"an exploratory configuration with separate coder and tester agents achieves the highest observed scores, with 97.6% Pass@3 and 88.8% Pass$^3$.","support":"source-stated"}],"limitations":["The authors evaluate a specific set of 17 finite element tasks; the abstract does not establish how results extend beyond those tasks.","The separate coder and tester configuration is described as exploratory, so its reported scores do not establish that it will improve results consistently across models and tasks."],"mediaId":"241a6a3a650e0124353c723a0ebb84934449a4839d362fc98686871ec20a4445","correctionReason":null,"id":"3f64330ddc5ccdac581903234c4b88468fc31d7aad99921d41ca6c3caea1eb4a","publicationId":"0f5f8624-7ed2-421d-8a16-c7e99a2e6b4f","version":1,"sourceUrl":"https://arxiv.org/abs/2611.21176","sourceTitle":"Scientific Coding with Large Language Models: Evidence from Finite Element Analysis","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-16","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-20T05:04:35.067Z","createdAt":"2026-11-20T05:34:54.460Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"241a6a3a650e0124353c723a0ebb84934449a4839d362fc98686871ec20a4445","kind":"article","url":"https://arxiv.org/abs/2611.21176","sourceUrl":"https://arxiv.org/abs/2611.21176","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-20T05:34:03.114Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.21176","checkedAt":"2026-11-20T22:38:09.785Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"241a6a3a650e0124353c723a0ebb84934449a4839d362fc98686871ec20a4445","confidence":0.64,"provider":"typesafe","model":"jev-1.13.0","inputHash":"206c298b8237213adec7b1dce4430c28223d8e47c146e951471f7168f9928c5f","decidedAt":"2026-11-20T05:34:05.068Z"},"provenance":{"textHash":"2766ab91bb9171705a6565362d03a5d1dc6fa3687c8a1ddf67abdc399c24f022","observationId":"bb217d34410c614803e495dca1d64a1936221e128826535a260d0e1c0d31a5df","retrievedAt":"2026-11-20T05:34:02.840Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"deaa029a-3bd2-4479-885d-d0d045a8065f"},"reviewer":{"model":"gpt-6-luna","runId":"b038498b-7b5f-4b15-839f-6abf69aca7e3"}},"audit":{"status":"passed","reviewedAt":"2026-11-20T05:34:54.460Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Hamdi, Erfan","url":null,"kind":"unknown"},{"name":"Shor, Joel","url":null,"kind":"unknown"},{"name":"Lejeune, Emma","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Hamdi, Erfan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Hamdi, Erfan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21176","locator":"head/meta[name=author|citation_author]"},"assertionId":"eb322beaba26500bc6915d1ac2d37d1a5ff509fe72b3a38879c4e227c99475f7:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-20T22:38:09.785Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"eb322beaba26500bc6915d1ac2d37d1a5ff509fe72b3a38879c4e227c99475f7","sourceUrl":"https://arxiv.org/abs/2611.21176","contentSha256":"5534b3e1a7af23eb47d275bb3bf9484f1c7f89d5e9d780bc4c37ec9cf56fe60b","textHash":"2766ab91bb9171705a6565362d03a5d1dc6fa3687c8a1ddf67abdc399c24f022","observedAt":"2026-11-20T22:38:09.785Z"}},{"actor":{"name":"Shor, Joel","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Shor, Joel","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21176","locator":"head/meta[name=author|citation_author]"},"assertionId":"eb322beaba26500bc6915d1ac2d37d1a5ff509fe72b3a38879c4e227c99475f7:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-20T22:38:09.785Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"eb322beaba26500bc6915d1ac2d37d1a5ff509fe72b3a38879c4e227c99475f7","sourceUrl":"https://arxiv.org/abs/2611.21176","contentSha256":"5534b3e1a7af23eb47d275bb3bf9484f1c7f89d5e9d780bc4c37ec9cf56fe60b","textHash":"2766ab91bb9171705a6565362d03a5d1dc6fa3687c8a1ddf67abdc399c24f022","observedAt":"2026-11-20T22:38:09.785Z"}},{"actor":{"name":"Lejeune, Emma","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Lejeune, Emma","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21176","locator":"head/meta[name=author|citation_author]"},"assertionId":"eb322beaba26500bc6915d1ac2d37d1a5ff509fe72b3a38879c4e227c99475f7:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-20T22:38:09.785Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"eb322beaba26500bc6915d1ac2d37d1a5ff509fe72b3a38879c4e227c99475f7","sourceUrl":"https://arxiv.org/abs/2611.21176","contentSha256":"5534b3e1a7af23eb47d275bb3bf9484f1c7f89d5e9d780bc4c37ec9cf56fe60b","textHash":"2766ab91bb9171705a6565362d03a5d1dc6fa3687c8a1ddf67abdc399c24f022","observedAt":"2026-11-20T22:38:09.785Z"}}],"participationCoverage":{"total":3,"returned":3,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.21176","snapshotId":"eb322beaba26500bc6915d1ac2d37d1a5ff509fe72b3a38879c4e227c99475f7","contentSha256":"5534b3e1a7af23eb47d275bb3bf9484f1c7f89d5e9d780bc4c37ec9cf56fe60b","textHash":"2766ab91bb9171705a6565362d03a5d1dc6fa3687c8a1ddf67abdc399c24f022","observedAt":"2026-11-20T22:38:09.785Z","basis":"publisher-page-metadata","sourceLinks":45,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/3f64330ddc5ccdac581903234c4b88468fc31d7aad99921d41ca6c3caea1eb4a/source"}}}],"audits":[{"publicationId":"0f5f8624-7ed2-421d-8a16-c7e99a2e6b4f","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading names Erfan Hamdi, Joel Shor and Emma Lejeune as the authors, which matches the arXiv listing. It frames the results as the authors' findings and argument rather than as established fact.","excerpt":"Authors:Erfan Hamdi, Joel Shor, Emma Lejeune"},{"dimension":"summary","status":"pass","explanation":"The reading accurately conveys the paper's thesis: consistency across attempts and independent verification, not one-off generation, are the main obstacles. It explains agentic harnesses, Pass@3 and Pass^3 in plain language and labels the coder/tester setup as exploratory. The title stresses verification but drops consistency, though the body restores it. Smallest optional repair: note that single-pass trials ran without execution feedback. Finite element is a technical term that should be explained by a tooltip. The question follows from the source.","excerpt":"These results identify consistency across attempts and independent verification as the main obstacles to trusting LLM-generated scientific code."},{"dimension":"date","status":"pass","explanation":"originalPublished is 2026-11-16, which matches the v1 submission date. The page shows a later v2 revision on 18 Nov and a feed date of 20 Nov, but the first submission date is the correct original publication date.","excerpt":"[Submitted on 16 Nov 2026 (v1), last revised 18 Nov 2026 (this version, v2)]"},{"dimension":"support","status":"pass","explanation":"Each figure is stated in the abstract: 87.1%, 96.5%/77.1%, the 249 trajectories, the 73.1% and 70.3% failure rates, and 97.6%/88.8%. So are the regressions and the shared errors between tests and implementations. Claim 2's excerpt covers only the ranking difference, but the regression point is stated elsewhere in the abstract. The limitations are appropriately cautious, and no unsupported benefits or impacts are implied.","excerpt":"Agentic execution solves some tasks that never succeeded in single-pass trials and improves consistency on others, but gains vary across models and harnesses, and regressions also occur."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct arXiv abstract page for this paper. It is the original source and the most direct way for readers to access it.","excerpt":null}],"createdAt":"2026-11-20T05:34:54.460Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}