# Measuring Where Intent Gets Lost in Agentic Smart Contract Synthesis [Canonical reading](https://escapemeridian.com/readings/measuring-where-intent-gets-lost-in-agentic-smart-contract-e8e3bd70) Authors Abhinav Goel, Agostino Capponi, Alfio Gliozzo and Chaitya Shah argue that software which runs may still fail to do what a client asked. That gap is especially consequential for smart contracts: programs behind Web3 applications that move money on a blockchain and cannot be fixed once deployed. The authors evaluate AI-written contracts by replaying the same transactions on them and on the original contracts they should reproduce. They report that common methods miss obvious mistakes, and that checking only functions shared by all systems hid about half the difference between pipelines. Their proposed elicitation-first architecture asks clients questions before code generation, checks the result against an independently written executable answer key, and returns disagreements to the client. Across three AI models, they report that asking first matters more than reviewing or patching afterwards; they also warn that automated security fixes often break requested behavior. The authors say a few well-chosen questions recover much of what a short description omits. The abstract does not give the task-level results or establish how well the approach works beyond the tested setting. How does the approach perform on contracts and client requirements outside the 600 released tasks? Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Oct 27, 2026 (2026-10-27). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-11T08:24:17.144Z. First observed: 2026-10-30T06:43:01.946Z. Retrieved: 2026-10-31T12:03:52.202Z. - [Measuring Where Intent Gets Lost in Agentic Smart Contract Synthesis](https://arxiv.org/abs/2610.37830) ## Limits The supplied evidence is the paper abstract, not its full methods or results; it does not provide task-level figures or detail the tested conditions. The abstract reports results across three AI models but does not establish performance on contracts or client requirements outside the study's 600 tasks. ## Open question How does the approach perform on contracts and client requirements outside the 600 released tasks? ## Source-stated The authors frame the central problem as a gap between code that runs and code that fulfills the request, with particular stakes for smart contracts that move money and cannot be fixed after deployment. > code that runs is not the same as code that does what was asked. This gap matters most for smart contracts - [Supporting source](https://arxiv.org/abs/2610.37830) ## Source-stated The authors evaluate synthesized contracts by replaying transactions against an AI-written contract and the original it should reproduce; they report that common methods miss obvious mistakes. > We replay the same transactions on an AI-written contract and on the original it should reproduce and measure how often they agree. We test every grading tool on planted mistakes - [Supporting source](https://arxiv.org/abs/2610.37830) ## Source-stated The authors report that checking only functions shared by every system concealed about half the difference between pipelines. > Checking only the functions every system includes hid about half the difference between pipelines. - [Supporting source](https://arxiv.org/abs/2610.37830) ## Source-stated The proposed elicitation-first architecture asks clients questions before generation and checks the result against an independently written executable answer key; the authors report asking first matters more than later review or patching across three AI models. > Before writing code, the agent asks the client a short checklist of questions, checks the contract against an independently written executable answer key - [Supporting source](https://arxiv.org/abs/2610.37830) ## Source-stated The authors caution that automated security fixes can break requested behavior and say they release tools, 600 tasks and all results. > automated security fixes often break requested behavior. A few well-chosen questions recover much of what a short description omits - [Supporting source](https://arxiv.org/abs/2610.37830) ## Source and contributor context Goel, Abhinav, Capponi, Agostino, Gliozzo, Alfio, Shah, Chaitya · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Agentic AI AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer. - [Definition reference](https://developers.google.com/machine-learning/glossary#agentic) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 1a3b93e1-e61b-4d4b-8470-06623e3a96ca. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-11T08:24:17.144Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 5d58d29ca12c160c2d9778527e48d62f66ba2649b204d3bce7fefec6e7378183. Source text hash: ada4ab328a1659e631142936bc5a5c6b45cd6dcddab268e978c635af3507a438. ## Revision and source-check history ### Version 1 — 2026-11-11T08:24:17.144Z Authors Abhinav Goel, Agostino Capponi, Alfio Gliozzo and Chaitya Shah argue that software which runs may still fail to do what a client asked. That gap is especially consequential for smart contracts: programs behind Web3 applications that move money on a blockchain and cannot be fixed once deployed. The authors evaluate AI-written contracts by replaying the same transactions on them and on the original contracts they should reproduce. They report that common methods miss obvious mistakes, and that checking only functions shared by all systems hid about half the difference between pipelines. Their proposed elicitation-first architecture asks clients questions before code generation, checks the result against an independently written executable answer key, and returns disagreements to the client. Across three AI models, they report that asking first matters more than reviewing or patching afterwards; they also warn that automated security fixes often break requested behavior. The authors say a few well-chosen questions recover much of what a short description omits. The abstract does not give the task-level results or establish how well the approach works beyond the tested setting. How does the approach perform on contracts and client requirements outside the 600 released tasks? ## Original source arXiv cs.AI · Article Original source date: Oct 27, 2026 (2026-10-27). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-11T08:24:17.144Z. First observed: 2026-10-30T06:43:01.946Z. Retrieved: 2026-10-31T12:03:52.202Z. - [Measuring Where Intent Gets Lost in Agentic Smart Contract Synthesis](https://arxiv.org/abs/2610.37830) ## Limits The supplied evidence is the paper abstract, not its full methods or results; it does not provide task-level figures or detail the tested conditions. The abstract reports results across three AI models but does not establish performance on contracts or client requirements outside the study's 600 tasks. ## Open question How does the approach perform on contracts and client requirements outside the 600 released tasks? ## Source-stated The authors frame the central problem as a gap between code that runs and code that fulfills the request, with particular stakes for smart contracts that move money and cannot be fixed after deployment. > code that runs is not the same as code that does what was asked. This gap matters most for smart contracts - [Supporting source](https://arxiv.org/abs/2610.37830) ## Source-stated The authors evaluate synthesized contracts by replaying transactions against an AI-written contract and the original it should reproduce; they report that common methods miss obvious mistakes. > We replay the same transactions on an AI-written contract and on the original it should reproduce and measure how often they agree. We test every grading tool on planted mistakes - [Supporting source](https://arxiv.org/abs/2610.37830) ## Source-stated The authors report that checking only functions shared by every system concealed about half the difference between pipelines. > Checking only the functions every system includes hid about half the difference between pipelines. - [Supporting source](https://arxiv.org/abs/2610.37830) ## Source-stated The proposed elicitation-first architecture asks clients questions before generation and checks the result against an independently written executable answer key; the authors report asking first matters more than later review or patching across three AI models. > Before writing code, the agent asks the client a short checklist of questions, checks the contract against an independently written executable answer key - [Supporting source](https://arxiv.org/abs/2610.37830) ## Source-stated The authors caution that automated security fixes can break requested behavior and say they release tools, 600 tasks and all results. > automated security fixes often break requested behavior. A few well-chosen questions recover much of what a short description omits - [Supporting source](https://arxiv.org/abs/2610.37830) ## Source and contributor context Goel, Abhinav, Capponi, Agostino, Gliozzo, Alfio, Shah, Chaitya · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Agentic AI AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer. - [Definition reference](https://developers.google.com/machine-learning/glossary#agentic) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 1a3b93e1-e61b-4d4b-8470-06623e3a96ca. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-11T08:24:17.144Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 5d58d29ca12c160c2d9778527e48d62f66ba2649b204d3bce7fefec6e7378183. Source text hash: ada4ab328a1659e631142936bc5a5c6b45cd6dcddab268e978c635af3507a438. ### passed — 2026-11-11T08:24:17.144Z — anthropic/claude-opus-5.5 Publication: 1a3b93e1-e61b-4d4b-8470-06623e3a96ca attribution — pass: The reading names the four listed authors and keeps the findings attributed to them with phrases such as 'they report' and 'the authors say.' It does not present the paper's claims as established facts or as Escape Meridian's view. > Authors:Abhinav Goel, Agostino Capponi, Alfio Gliozzo, Chaitya Shah summary — pass: A general reader can follow the reading. It leads with the gap between code that runs and code that does what was asked, explains smart contracts, and describes the method and the ask-first proposal. Two small fidelity repairs: the source says asking first matters 'far more,' which the reading softens to 'more'; and it omits 'instead of silently rewriting code.' 'Pipelines' could use a tooltip. The question follows from the stated 600 tasks and is not leading. > Asking first matters far more than reviewing or patching afterwards, across three AI models date — pass: originalPublished is 2026-10-27, which matches the arXiv v1 submission date. The later feed date of Oct 31 is correctly not used. > \[Submitted on 27 Oct 2026\] support — pass: All claims are stated in the abstract. Two claim excerpts do not cover their full claim. Claim 2's excerpt omits the 'miss obvious mistakes' line, and claim 5's excerpt omits the release of tools and 600 tasks. Both statements appear elsewhere in the source. Claim 5 says security fixes 'can' break behavior, while the source says 'often'; the summary text preserves 'often.' The limitations are appropriately framed. > Common methods, such as AI judges and standard analysis tools, miss obvious mistakes media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most direct access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/e8e3bd7060d2a0ca1809542779c19776849b38e499a721db539784483fd4bc87/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/measuring-where-intent-gets-lost-in-agentic-smart-contract-e8e3bd70.json)