Khati, Dipin, Mehrotra, Shubham, Bi, Bin, Yu, Zhou, Poshyvanyk, Denys, Zhu, James, Asur, Sitaram, Mui, Phil L. · Nov 9, 2026
arXiv cs.AI: AgentScript-Eval benchmarks enterprise DSL code generation
Markdown · JSON · Original source
The authors of AgentScript-Eval argue that success at general-purpose code generation does not ensure success with enterprise DSLs, which are less represented in public data. They introduce an execution-graded of 100 AgentScript tasks, covering agent authoring, debugging and optimization. In tests of five models, the best reached 25.0% pass@3, while every model scored 0% on authoring from scratch. The authors report that model size and price were weak predictors: a 31B had the highest compilation rate and matched the strongest on partial-credit accuracy at the lowest token cost. An compile-and-edit loop roughly doubled partial-credit accuracy. Guidance also mattered: the same skill pack scored 28% as plain documentation and 14% as a live plugin; rewriting one skill in the plugin raised the score to 37%, the paper’s best result. The authors’ central point is that, in these tests, the setup around a model and the guidance it receives mattered more than model size. The findings concern this benchmark; the supplied abstract does not establish whether they carry over to other enterprise DSLs or tasks.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 9, 2026 (2026-11-09).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-11T23:16:32.390Z. First observed: 2026-11-11T05:03:59.328Z. Retrieved: 2026-11-11T23:04:55.738Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The reported evaluation covers 100 AgentScript tasks and five models; the abstract does not establish whether the results generalize to other enterprise DSLs or tasks. The supplied evidence is the paper abstract, not its full methods or results.
Open question
Would the reported results hold for enterprise DSLs or tasks beyond the AgentScript ?
Source-stated
The authors introduce an execution-graded benchmark of 100 AgentScript tasks covering authoring, debugging and optimization.
We introduce AgentScript-Eval, an execution-graded benchmark of 100 real tasks for AgentScript
Source-stated
In zero-shot evaluation of five models, the best achieved 25.0% pass@3, and all models scored 0% on authoring from scratch.
Evaluating five models zero-shot, we find that the regime is hard: the best model reaches only 25.0% pass@3, and every model scores 0% on authoring from scratch.
Source-stated
The authors report model scale and price were weak predictors; a 31B open-weight model had the highest compilation rate and matched the strongest frontier model on partial-credit accuracy at the lowest token cost.
a 31B open-weight model attains the highest compilation rate of any model tested (48.0%) and matches the strongest frontier model on partial-credit accuracy at the lowest token cost.
Source-stated
The authors report that an agentic compile-and-edit loop roughly doubled partial-credit accuracy.
Adding an agentic compile-and-edit loop is the single largest lever, roughly doubling partial-credit accuracy.
Source-stated
The authors report different results for the same skill pack in different forms, and a higher score after rewriting one skill in the plugin.
The same skill pack scores 28% as plain documentation but only 14% as a live plugin. Rewriting one skill inside that plugin then raises it to 37%
Source-stated
In this benchmark, the authors conclude that the harness and supplied guidance mattered more than model size.
what mattered was the harness surrounding the model and the guidance supplied to it.
Source and contributor context
Khati, Dipin, Mehrotra, Shubham, Bi, Bin, Yu, Zhou, Poshyvanyk, Denys, Zhu, James, Asur, Sitaram, Mui, Phil L. · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Pre-training
A model's initial training on a large dataset, before further adaptation for particular uses.
Zero-shot
Doing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).
Open weights
A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
Agentic AI
AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: b8b741a4-6be3-4792-810e-db2047bc036d.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T23:16:32.390Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 13cd2c8401b0acccb610c3787dbf6d384b57c0c6f65ed17c04e423dbb9b386d8. Source text hash: e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517.
Revision and source-check history
Version 1 · 2026-11-11T23:16:32.390Z
arXiv cs.AI: AgentScript-Eval benchmarks enterprise DSL code generation
The authors of AgentScript-Eval argue that success at general-purpose code generation does not ensure success with enterprise DSLs, which are less represented in public data. They introduce an execution-graded of 100 AgentScript tasks, covering agent authoring, debugging and optimization. In tests of five models, the best reached 25.0% pass@3, while every model scored 0% on authoring from scratch. The authors report that model size and price were weak predictors: a 31B had the highest compilation rate and matched the strongest on partial-credit accuracy at the lowest token cost. An compile-and-edit loop roughly doubled partial-credit accuracy. Guidance also mattered: the same skill pack scored 28% as plain documentation and 14% as a live plugin; rewriting one skill in the plugin raised the score to 37%, the paper’s best result. The authors’ central point is that, in these tests, the setup around a model and the guidance it receives mattered more than model size. The findings concern this benchmark; the supplied abstract does not establish whether they carry over to other enterprise DSLs or tasks.
Original source
cs.AI · Article
Source details
Original source date: Nov 9, 2026 (2026-11-09).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-11T23:16:32.390Z. First observed: 2026-11-11T05:03:59.328Z. Retrieved: 2026-11-11T23:04:55.738Z.
Limits
The reported evaluation covers 100 AgentScript tasks and five models; the abstract does not establish whether the results generalize to other enterprise DSLs or tasks. The supplied evidence is the paper abstract, not its full methods or results.
Open question
Would the reported results hold for enterprise DSLs or tasks beyond the AgentScript ?
Source-stated
The authors introduce an execution-graded benchmark of 100 AgentScript tasks covering authoring, debugging and optimization.
We introduce AgentScript-Eval, an execution-graded benchmark of 100 real tasks for AgentScript
Source-stated
In zero-shot evaluation of five models, the best achieved 25.0% pass@3, and all models scored 0% on authoring from scratch.
Evaluating five models zero-shot, we find that the regime is hard: the best model reaches only 25.0% pass@3, and every model scores 0% on authoring from scratch.
Source-stated
The authors report model scale and price were weak predictors; a 31B open-weight model had the highest compilation rate and matched the strongest frontier model on partial-credit accuracy at the lowest token cost.
a 31B open-weight model attains the highest compilation rate of any model tested (48.0%) and matches the strongest frontier model on partial-credit accuracy at the lowest token cost.
Source-stated
The authors report that an agentic compile-and-edit loop roughly doubled partial-credit accuracy.
Adding an agentic compile-and-edit loop is the single largest lever, roughly doubling partial-credit accuracy.
Source-stated
The authors report different results for the same skill pack in different forms, and a higher score after rewriting one skill in the plugin.
The same skill pack scores 28% as plain documentation but only 14% as a live plugin. Rewriting one skill inside that plugin then raises it to 37%
Source-stated
In this benchmark, the authors conclude that the harness and supplied guidance mattered more than model size.
what mattered was the harness surrounding the model and the guidance supplied to it.
Source and contributor context
Khati, Dipin, Mehrotra, Shubham, Bi, Bin, Yu, Zhou, Poshyvanyk, Denys, Zhu, James, Asur, Sitaram, Mui, Phil L. · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Pre-training
A model's initial training on a large dataset, before further adaptation for particular uses.
Zero-shot
Doing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).
Open weights
A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
Agentic AI
AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: b8b741a4-6be3-4792-810e-db2047bc036d.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T23:16:32.390Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 13cd2c8401b0acccb610c3787dbf6d384b57c0c6f65ed17c04e423dbb9b386d8. Source text hash: e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517.
uncertain · 2026-11-11T23:16:32.390Z · anthropic/claude-opus-5.5
Publication: b8b741a4-6be3-4792-810e-db2047bc036d
attribution — pass: The findings and conclusions are consistently attributed to the paper's authors, and the arXiv source is identified correctly. The 'arXiv cs.AI' label matches the supplied publisher and the paper's cross-listing, although its primary category is cs.SE.
Authors:Dipin Khati, Shubham Mehrotra, Bin Bi, Zhou Yu, Denys Poshyvanyk, James Zhu, Sitaram Asur, Phil L. Mui
summary — uncertain: The summary is faithful, but a general reader cannot follow its central terms. It never explains DSL (domain-specific language) or what AgentScript is: the declarative language Salesforce uses in Agentforce to build conversational agents. The terms pass@3, zero-shot and compilation rate are also unexplained. Smallest repair: spell out DSL, and identify AgentScript as Salesforce's Agentforce agent-definition language. Then gloss pass@3 and zero-shot via tooltips; add them to the term library if missing.
AgentScript, the declarative DSL that Salesforce uses in Agentforce to build conversational agents
date — pass: The originalPublished value of 2026-11-09 matches the v1 submission date shown on the arXiv page.
[Submitted on 9 Nov 2026]
support — pass: Every claim and number matches the abstract: 25.0% pass@3, 0% on authoring from scratch, the 31B model's 48.0% compilation rate, roughly doubled accuracy from the loop, and the 28/14/37% skill-pack results. The draft keeps the scope limitation and does not overstate generalization.
Larger models did not reliably outperform smaller ones; what mattered was the harness surrounding the model and the guidance supplied to it.
media — pass: The selected media is the verified, direct link to the original arXiv abstract page.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A model's initial training on a large dataset, before further adaptation for particular uses.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceDoing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).
Definition referenceA model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceAI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA model's initial training on a large dataset, before further adaptation for particular uses.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceDoing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).
Definition referenceA model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceAI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference