{"id":"108cfa19bf7f0c7bdf033323b224dcae237ee630ff3b6f11fb0ba1adb215dd6d","title":"arXiv cs.AI: AgentScript-Eval benchmarks enterprise DSL code generation","argument":"The authors of AgentScript-Eval argue that success at general-purpose code generation does not ensure success with enterprise DSLs, which are less represented in public pretraining data. They introduce an execution-graded benchmark of 100 AgentScript tasks, covering agent authoring, debugging and optimization. In zero-shot tests of five models, the best reached 25.0% pass@3, while every model scored 0% on authoring from scratch. The authors report that model size and price were weak predictors: a 31B open-weight model had the highest compilation rate and matched the strongest frontier model on partial-credit accuracy at the lowest token cost. An agentic compile-and-edit loop roughly doubled partial-credit accuracy. Guidance also mattered: the same skill pack scored 28% as plain documentation and 14% as a live plugin; rewriting one skill in the plugin raised the score to 37%, the paper’s best result. The authors’ central point is that, in these tests, the setup around a model and the guidance it receives mattered more than model size. The findings concern this benchmark; the supplied abstract does not establish whether they carry over to other enterprise DSLs or tasks.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported evaluation covers 100 AgentScript tasks and five models; the abstract does not establish whether the results generalize to other enterprise DSLs or tasks. The supplied evidence is the paper abstract, not its full methods or results.","speaker":{"name":"Khati, Dipin, Mehrotra, Shubham, Bi, Bin, Yu, Zhou, Poshyvanyk, Denys, Zhu, James, Asur, Sitaram, Mui, Phil L.","roleAtSource":"Article author"},"source":{"title":"AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSL","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.13183","publishedAt":"2026-11-09","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce an execution-graded benchmark of 100 AgentScript tasks covering authoring, debugging and optimization.","excerpt":"We introduce AgentScript-Eval, an execution-graded benchmark of 100 real tasks for AgentScript","url":"https://arxiv.org/abs/2611.13183","relationship":"Source-stated"},{"text":"In zero-shot evaluation of five models, the best achieved 25.0% pass@3, and all models scored 0% on authoring from scratch.","excerpt":"Evaluating five models zero-shot, we find that the regime is hard: the best model reaches only 25.0% pass@3, and every model scores 0% on authoring from scratch.","url":"https://arxiv.org/abs/2611.13183","relationship":"Source-stated"},{"text":"The authors report model scale and price were weak predictors; a 31B open-weight model had the highest compilation rate and matched the strongest frontier model on partial-credit accuracy at the lowest token cost.","excerpt":"a 31B open-weight model attains the highest compilation rate of any model tested (48.0%) and matches the strongest frontier model on partial-credit accuracy at the lowest token cost.","url":"https://arxiv.org/abs/2611.13183","relationship":"Source-stated"},{"text":"The authors report that an agentic compile-and-edit loop roughly doubled partial-credit accuracy.","excerpt":"Adding an agentic compile-and-edit loop is the single largest lever, roughly doubling partial-credit accuracy.","url":"https://arxiv.org/abs/2611.13183","relationship":"Source-stated"},{"text":"The authors report different results for the same skill pack in different forms, and a higher score after rewriting one skill in the plugin.","excerpt":"The same skill pack scores 28% as plain documentation but only 14% as a live plugin. Rewriting one skill inside that plugin then raises it to 37%","url":"https://arxiv.org/abs/2611.13183","relationship":"Source-stated"},{"text":"In this benchmark, the authors conclude that the harness and supplied guidance mattered more than model size.","excerpt":"what mattered was the harness surrounding the model and the guidance supplied to it.","url":"https://arxiv.org/abs/2611.13183","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.13183","checkedAt":"2026-11-11T23:48:22.265Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Would the reported results hold for enterprise DSLs or tasks beyond the AgentScript benchmark?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: AgentScript-Eval benchmarks enterprise DSL code generation","summary":"The authors of AgentScript-Eval argue that success at general-purpose code generation does not ensure success with enterprise DSLs, which are less represented in public pretraining data. They introduce an execution-graded benchmark of 100 AgentScript tasks, covering agent authoring, debugging and optimization. In zero-shot tests of five models, the best reached 25.0% pass@3, while every model scored 0% on authoring from scratch. The authors report that model size and price were weak predictors: a 31B open-weight model had the highest compilation rate and matched the strongest frontier model on partial-credit accuracy at the lowest token cost. An agentic compile-and-edit loop roughly doubled partial-credit accuracy. Guidance also mattered: the same skill pack scored 28% as plain documentation and 14% as a live plugin; rewriting one skill in the plugin raised the score to 37%, the paper’s best result. The authors’ central point is that, in these tests, the setup around a model and the guidance it receives mattered more than model size. The findings concern this benchmark; the supplied abstract does not establish whether they carry over to other enterprise DSLs or tasks.","question":"Would the reported results hold for enterprise DSLs or tasks beyond the AgentScript benchmark?","claims":[{"text":"The authors introduce an execution-graded benchmark of 100 AgentScript tasks covering authoring, debugging and optimization.","excerpt":"We introduce AgentScript-Eval, an execution-graded benchmark of 100 real tasks for AgentScript","support":"source-stated"},{"text":"In zero-shot evaluation of five models, the best achieved 25.0% pass@3, and all models scored 0% on authoring from scratch.","excerpt":"Evaluating five models zero-shot, we find that the regime is hard: the best model reaches only 25.0% pass@3, and every model scores 0% on authoring from scratch.","support":"source-stated"},{"text":"The authors report model scale and price were weak predictors; a 31B open-weight model had the highest compilation rate and matched the strongest frontier model on partial-credit accuracy at the lowest token cost.","excerpt":"a 31B open-weight model attains the highest compilation rate of any model tested (48.0%) and matches the strongest frontier model on partial-credit accuracy at the lowest token cost.","support":"source-stated"},{"text":"The authors report that an agentic compile-and-edit loop roughly doubled partial-credit accuracy.","excerpt":"Adding an agentic compile-and-edit loop is the single largest lever, roughly doubling partial-credit accuracy.","support":"source-stated"},{"text":"The authors report different results for the same skill pack in different forms, and a higher score after rewriting one skill in the plugin.","excerpt":"The same skill pack scores 28% as plain documentation but only 14% as a live plugin. Rewriting one skill inside that plugin then raises it to 37%","support":"source-stated"},{"text":"In this benchmark, the authors conclude that the harness and supplied guidance mattered more than model size.","excerpt":"what mattered was the harness surrounding the model and the guidance supplied to it.","support":"source-stated"}],"limitations":["The reported evaluation covers 100 AgentScript tasks and five models; the abstract does not establish whether the results generalize to other enterprise DSLs or tasks.","The supplied evidence is the paper abstract, not its full methods or results."],"mediaId":"7cd44bcff16a11e30198e166aed39a295aea86df3188b074ddd16e9fd0658be7","correctionReason":null,"id":"108cfa19bf7f0c7bdf033323b224dcae237ee630ff3b6f11fb0ba1adb215dd6d","publicationId":"b8b741a4-6be3-4792-810e-db2047bc036d","version":1,"sourceUrl":"https://arxiv.org/abs/2611.13183","sourceTitle":"AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSL","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-09","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-11T05:03:59.328Z","createdAt":"2026-11-11T23:16:32.390Z","historical":false,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"7cd44bcff16a11e30198e166aed39a295aea86df3188b074ddd16e9fd0658be7","kind":"article","url":"https://arxiv.org/abs/2611.13183","sourceUrl":"https://arxiv.org/abs/2611.13183","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-11T23:04:56.038Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.13183","checkedAt":"2026-11-11T23:48:22.265Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"7cd44bcff16a11e30198e166aed39a295aea86df3188b074ddd16e9fd0658be7","confidence":0.89,"provider":"typesafe","model":"jev-1.13.0","inputHash":"c530906d243770732baefdae0481a8922b17c064693780aa24d12b5275b2b27b","decidedAt":"2026-11-11T23:04:58.162Z"},"provenance":{"textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observationId":"13cd2c8401b0acccb610c3787dbf6d384b57c0c6f65ed17c04e423dbb9b386d8","retrievedAt":"2026-11-11T23:04:55.738Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"d681902e-8564-4798-871c-686fd203ff7a"},"reviewer":{"model":"gpt-6-luna","runId":"01f15dda-d616-43cd-8f5b-7deab4f1425f"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-11T23:16:32.390Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Khati, Dipin","url":null,"kind":"unknown"},{"name":"Mehrotra, Shubham","url":null,"kind":"unknown"},{"name":"Bi, Bin","url":null,"kind":"unknown"},{"name":"Yu, Zhou","url":null,"kind":"unknown"},{"name":"Poshyvanyk, Denys","url":null,"kind":"unknown"},{"name":"Zhu, James","url":null,"kind":"unknown"},{"name":"Asur, Sitaram","url":null,"kind":"unknown"},{"name":"Mui, Phil L.","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Khati, Dipin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Khati, Dipin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13183","locator":"head/meta[name=author|citation_author]"},"assertionId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T23:48:22.265Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368","sourceUrl":"https://arxiv.org/abs/2611.13183","contentSha256":"12f4b34a300f0a7439f5cbf9a29e3fabc778a7fb5ea80631627d1000ab7204f6","textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observedAt":"2026-11-11T23:48:22.265Z"}},{"actor":{"name":"Mehrotra, Shubham","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mehrotra, Shubham","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13183","locator":"head/meta[name=author|citation_author]"},"assertionId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T23:48:22.265Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368","sourceUrl":"https://arxiv.org/abs/2611.13183","contentSha256":"12f4b34a300f0a7439f5cbf9a29e3fabc778a7fb5ea80631627d1000ab7204f6","textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observedAt":"2026-11-11T23:48:22.265Z"}},{"actor":{"name":"Bi, Bin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Bi, Bin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13183","locator":"head/meta[name=author|citation_author]"},"assertionId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T23:48:22.265Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368","sourceUrl":"https://arxiv.org/abs/2611.13183","contentSha256":"12f4b34a300f0a7439f5cbf9a29e3fabc778a7fb5ea80631627d1000ab7204f6","textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observedAt":"2026-11-11T23:48:22.265Z"}},{"actor":{"name":"Yu, Zhou","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yu, Zhou","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13183","locator":"head/meta[name=author|citation_author]"},"assertionId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T23:48:22.265Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368","sourceUrl":"https://arxiv.org/abs/2611.13183","contentSha256":"12f4b34a300f0a7439f5cbf9a29e3fabc778a7fb5ea80631627d1000ab7204f6","textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observedAt":"2026-11-11T23:48:22.265Z"}},{"actor":{"name":"Poshyvanyk, Denys","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Poshyvanyk, Denys","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13183","locator":"head/meta[name=author|citation_author]"},"assertionId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T23:48:22.265Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368","sourceUrl":"https://arxiv.org/abs/2611.13183","contentSha256":"12f4b34a300f0a7439f5cbf9a29e3fabc778a7fb5ea80631627d1000ab7204f6","textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observedAt":"2026-11-11T23:48:22.265Z"}},{"actor":{"name":"Zhu, James","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhu, James","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13183","locator":"head/meta[name=author|citation_author]"},"assertionId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T23:48:22.265Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368","sourceUrl":"https://arxiv.org/abs/2611.13183","contentSha256":"12f4b34a300f0a7439f5cbf9a29e3fabc778a7fb5ea80631627d1000ab7204f6","textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observedAt":"2026-11-11T23:48:22.265Z"}},{"actor":{"name":"Asur, Sitaram","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Asur, Sitaram","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13183","locator":"head/meta[name=author|citation_author]"},"assertionId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T23:48:22.265Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368","sourceUrl":"https://arxiv.org/abs/2611.13183","contentSha256":"12f4b34a300f0a7439f5cbf9a29e3fabc778a7fb5ea80631627d1000ab7204f6","textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observedAt":"2026-11-11T23:48:22.265Z"}},{"actor":{"name":"Mui, Phil L.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mui, Phil L.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13183","locator":"head/meta[name=author|citation_author]"},"assertionId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T23:48:22.265Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368","sourceUrl":"https://arxiv.org/abs/2611.13183","contentSha256":"12f4b34a300f0a7439f5cbf9a29e3fabc778a7fb5ea80631627d1000ab7204f6","textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observedAt":"2026-11-11T23:48:22.265Z"}}],"participationCoverage":{"total":8,"returned":8,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.13183","snapshotId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368","contentSha256":"12f4b34a300f0a7439f5cbf9a29e3fabc778a7fb5ea80631627d1000ab7204f6","textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observedAt":"2026-11-11T23:48:22.265Z","basis":"publisher-page-metadata","sourceLinks":47,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/108cfa19bf7f0c7bdf033323b224dcae237ee630ff3b6f11fb0ba1adb215dd6d/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-agentscript-eval-benchmarks-enterprise-dsl-code-108cfa19","formats":{"html":"/readings/arxiv-cs-ai-agentscript-eval-benchmarks-enterprise-dsl-code-108cfa19","markdown":"/readings/arxiv-cs-ai-agentscript-eval-benchmarks-enterprise-dsl-code-108cfa19.md","json":"/readings/arxiv-cs-ai-agentscript-eval-benchmarks-enterprise-dsl-code-108cfa19.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-pre-training","name":"Pre-training","explanation":"A model's initial training on a large dataset, before further adaptation for particular uses.","referenceUrl":"https://developers.google.com/machine-learning/glossary#pre-training"},{"id":"term-zero-shot","name":"Zero-shot","explanation":"Doing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).","referenceUrl":"https://developers.google.com/machine-learning/glossary#zero-shot-prompting"},{"id":"term-open-weights","name":"Open weights","explanation":"A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.","referenceUrl":"https://opensource.org/ai/open-weights"},{"id":"term-frontier-model","name":"Frontier model","explanation":"One of the most capable AI models available at a given time; the set shifts as models improve.","referenceUrl":"https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper"},{"id":"term-agentic","name":"Agentic AI","explanation":"AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.","referenceUrl":"https://developers.google.com/machine-learning/glossary#agentic"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"108cfa19bf7f0c7bdf033323b224dcae237ee630ff3b6f11fb0ba1adb215dd6d","revision":"b8b741a4-6be3-4792-810e-db2047bc036d","revisionKind":"publication-id","title":"arXiv cs.AI: AgentScript-Eval benchmarks enterprise DSL code generation","url":"https://escapemeridian.com/readings/arxiv-cs-ai-agentscript-eval-benchmarks-enterprise-dsl-code-108cfa19"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.13183","revision":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","revisionKind":"sha256-source-text","title":"AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSL","url":"https://arxiv.org/abs/2611.13183"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"108cfa19bf7f0c7bdf033323b224dcae237ee630ff3b6f11fb0ba1adb215dd6d","title":"arXiv cs.AI: AgentScript-Eval benchmarks enterprise DSL code generation","argument":"The authors of AgentScript-Eval argue that success at general-purpose code generation does not ensure success with enterprise DSLs, which are less represented in public pretraining data. They introduce an execution-graded benchmark of 100 AgentScript tasks, covering agent authoring, debugging and optimization. In zero-shot tests of five models, the best reached 25.0% pass@3, while every model scored 0% on authoring from scratch. The authors report that model size and price were weak predictors: a 31B open-weight model had the highest compilation rate and matched the strongest frontier model on partial-credit accuracy at the lowest token cost. An agentic compile-and-edit loop roughly doubled partial-credit accuracy. Guidance also mattered: the same skill pack scored 28% as plain documentation and 14% as a live plugin; rewriting one skill in the plugin raised the score to 37%, the paper’s best result. The authors’ central point is that, in these tests, the setup around a model and the guidance it receives mattered more than model size. The findings concern this benchmark; the supplied abstract does not establish whether they carry over to other enterprise DSLs or tasks.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported evaluation covers 100 AgentScript tasks and five models; the abstract does not establish whether the results generalize to other enterprise DSLs or tasks. The supplied evidence is the paper abstract, not its full methods or results.","speaker":{"name":"Khati, Dipin, Mehrotra, Shubham, Bi, Bin, Yu, Zhou, Poshyvanyk, Denys, Zhu, James, Asur, Sitaram, Mui, Phil L.","roleAtSource":"Article author"},"source":{"title":"AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSL","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.13183","publishedAt":"2026-11-09","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce an execution-graded benchmark of 100 AgentScript tasks covering authoring, debugging and optimization.","excerpt":"We introduce AgentScript-Eval, an execution-graded benchmark of 100 real tasks for AgentScript","url":"https://arxiv.org/abs/2611.13183","relationship":"Source-stated"},{"text":"In zero-shot evaluation of five models, the best achieved 25.0% pass@3, and all models scored 0% on authoring from scratch.","excerpt":"Evaluating five models zero-shot, we find that the regime is hard: the best model reaches only 25.0% pass@3, and every model scores 0% on authoring from scratch.","url":"https://arxiv.org/abs/2611.13183","relationship":"Source-stated"},{"text":"The authors report model scale and price were weak predictors; a 31B open-weight model had the highest compilation rate and matched the strongest frontier model on partial-credit accuracy at the lowest token cost.","excerpt":"a 31B open-weight model attains the highest compilation rate of any model tested (48.0%) and matches the strongest frontier model on partial-credit accuracy at the lowest token cost.","url":"https://arxiv.org/abs/2611.13183","relationship":"Source-stated"},{"text":"The authors report that an agentic compile-and-edit loop roughly doubled partial-credit accuracy.","excerpt":"Adding an agentic compile-and-edit loop is the single largest lever, roughly doubling partial-credit accuracy.","url":"https://arxiv.org/abs/2611.13183","relationship":"Source-stated"},{"text":"The authors report different results for the same skill pack in different forms, and a higher score after rewriting one skill in the plugin.","excerpt":"The same skill pack scores 28% as plain documentation but only 14% as a live plugin. Rewriting one skill inside that plugin then raises it to 37%","url":"https://arxiv.org/abs/2611.13183","relationship":"Source-stated"},{"text":"In this benchmark, the authors conclude that the harness and supplied guidance mattered more than model size.","excerpt":"what mattered was the harness surrounding the model and the guidance supplied to it.","url":"https://arxiv.org/abs/2611.13183","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.13183","checkedAt":"2026-11-11T23:48:22.265Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Would the reported results hold for enterprise DSLs or tasks beyond the AgentScript benchmark?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: AgentScript-Eval benchmarks enterprise DSL code generation","summary":"The authors of AgentScript-Eval argue that success at general-purpose code generation does not ensure success with enterprise DSLs, which are less represented in public pretraining data. They introduce an execution-graded benchmark of 100 AgentScript tasks, covering agent authoring, debugging and optimization. In zero-shot tests of five models, the best reached 25.0% pass@3, while every model scored 0% on authoring from scratch. The authors report that model size and price were weak predictors: a 31B open-weight model had the highest compilation rate and matched the strongest frontier model on partial-credit accuracy at the lowest token cost. An agentic compile-and-edit loop roughly doubled partial-credit accuracy. Guidance also mattered: the same skill pack scored 28% as plain documentation and 14% as a live plugin; rewriting one skill in the plugin raised the score to 37%, the paper’s best result. The authors’ central point is that, in these tests, the setup around a model and the guidance it receives mattered more than model size. The findings concern this benchmark; the supplied abstract does not establish whether they carry over to other enterprise DSLs or tasks.","question":"Would the reported results hold for enterprise DSLs or tasks beyond the AgentScript benchmark?","claims":[{"text":"The authors introduce an execution-graded benchmark of 100 AgentScript tasks covering authoring, debugging and optimization.","excerpt":"We introduce AgentScript-Eval, an execution-graded benchmark of 100 real tasks for AgentScript","support":"source-stated"},{"text":"In zero-shot evaluation of five models, the best achieved 25.0% pass@3, and all models scored 0% on authoring from scratch.","excerpt":"Evaluating five models zero-shot, we find that the regime is hard: the best model reaches only 25.0% pass@3, and every model scores 0% on authoring from scratch.","support":"source-stated"},{"text":"The authors report model scale and price were weak predictors; a 31B open-weight model had the highest compilation rate and matched the strongest frontier model on partial-credit accuracy at the lowest token cost.","excerpt":"a 31B open-weight model attains the highest compilation rate of any model tested (48.0%) and matches the strongest frontier model on partial-credit accuracy at the lowest token cost.","support":"source-stated"},{"text":"The authors report that an agentic compile-and-edit loop roughly doubled partial-credit accuracy.","excerpt":"Adding an agentic compile-and-edit loop is the single largest lever, roughly doubling partial-credit accuracy.","support":"source-stated"},{"text":"The authors report different results for the same skill pack in different forms, and a higher score after rewriting one skill in the plugin.","excerpt":"The same skill pack scores 28% as plain documentation but only 14% as a live plugin. Rewriting one skill inside that plugin then raises it to 37%","support":"source-stated"},{"text":"In this benchmark, the authors conclude that the harness and supplied guidance mattered more than model size.","excerpt":"what mattered was the harness surrounding the model and the guidance supplied to it.","support":"source-stated"}],"limitations":["The reported evaluation covers 100 AgentScript tasks and five models; the abstract does not establish whether the results generalize to other enterprise DSLs or tasks.","The supplied evidence is the paper abstract, not its full methods or results."],"mediaId":"7cd44bcff16a11e30198e166aed39a295aea86df3188b074ddd16e9fd0658be7","correctionReason":null,"id":"108cfa19bf7f0c7bdf033323b224dcae237ee630ff3b6f11fb0ba1adb215dd6d","publicationId":"b8b741a4-6be3-4792-810e-db2047bc036d","version":1,"sourceUrl":"https://arxiv.org/abs/2611.13183","sourceTitle":"AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSL","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-09","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-11T05:03:59.328Z","createdAt":"2026-11-11T23:16:32.390Z","historical":false,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"7cd44bcff16a11e30198e166aed39a295aea86df3188b074ddd16e9fd0658be7","kind":"article","url":"https://arxiv.org/abs/2611.13183","sourceUrl":"https://arxiv.org/abs/2611.13183","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-11T23:04:56.038Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.13183","checkedAt":"2026-11-11T23:48:22.265Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"7cd44bcff16a11e30198e166aed39a295aea86df3188b074ddd16e9fd0658be7","confidence":0.89,"provider":"typesafe","model":"jev-1.13.0","inputHash":"c530906d243770732baefdae0481a8922b17c064693780aa24d12b5275b2b27b","decidedAt":"2026-11-11T23:04:58.162Z"},"provenance":{"textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observationId":"13cd2c8401b0acccb610c3787dbf6d384b57c0c6f65ed17c04e423dbb9b386d8","retrievedAt":"2026-11-11T23:04:55.738Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"d681902e-8564-4798-871c-686fd203ff7a"},"reviewer":{"model":"gpt-6-luna","runId":"01f15dda-d616-43cd-8f5b-7deab4f1425f"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-11T23:16:32.390Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Khati, Dipin","url":null,"kind":"unknown"},{"name":"Mehrotra, Shubham","url":null,"kind":"unknown"},{"name":"Bi, Bin","url":null,"kind":"unknown"},{"name":"Yu, Zhou","url":null,"kind":"unknown"},{"name":"Poshyvanyk, Denys","url":null,"kind":"unknown"},{"name":"Zhu, James","url":null,"kind":"unknown"},{"name":"Asur, Sitaram","url":null,"kind":"unknown"},{"name":"Mui, Phil L.","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Khati, Dipin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Khati, Dipin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13183","locator":"head/meta[name=author|citation_author]"},"assertionId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T23:48:22.265Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368","sourceUrl":"https://arxiv.org/abs/2611.13183","contentSha256":"12f4b34a300f0a7439f5cbf9a29e3fabc778a7fb5ea80631627d1000ab7204f6","textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observedAt":"2026-11-11T23:48:22.265Z"}},{"actor":{"name":"Mehrotra, Shubham","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mehrotra, Shubham","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13183","locator":"head/meta[name=author|citation_author]"},"assertionId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T23:48:22.265Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368","sourceUrl":"https://arxiv.org/abs/2611.13183","contentSha256":"12f4b34a300f0a7439f5cbf9a29e3fabc778a7fb5ea80631627d1000ab7204f6","textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observedAt":"2026-11-11T23:48:22.265Z"}},{"actor":{"name":"Bi, Bin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Bi, Bin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13183","locator":"head/meta[name=author|citation_author]"},"assertionId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T23:48:22.265Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368","sourceUrl":"https://arxiv.org/abs/2611.13183","contentSha256":"12f4b34a300f0a7439f5cbf9a29e3fabc778a7fb5ea80631627d1000ab7204f6","textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observedAt":"2026-11-11T23:48:22.265Z"}},{"actor":{"name":"Yu, Zhou","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yu, Zhou","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13183","locator":"head/meta[name=author|citation_author]"},"assertionId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T23:48:22.265Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368","sourceUrl":"https://arxiv.org/abs/2611.13183","contentSha256":"12f4b34a300f0a7439f5cbf9a29e3fabc778a7fb5ea80631627d1000ab7204f6","textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observedAt":"2026-11-11T23:48:22.265Z"}},{"actor":{"name":"Poshyvanyk, Denys","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Poshyvanyk, Denys","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13183","locator":"head/meta[name=author|citation_author]"},"assertionId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T23:48:22.265Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368","sourceUrl":"https://arxiv.org/abs/2611.13183","contentSha256":"12f4b34a300f0a7439f5cbf9a29e3fabc778a7fb5ea80631627d1000ab7204f6","textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observedAt":"2026-11-11T23:48:22.265Z"}},{"actor":{"name":"Zhu, James","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhu, James","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13183","locator":"head/meta[name=author|citation_author]"},"assertionId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T23:48:22.265Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368","sourceUrl":"https://arxiv.org/abs/2611.13183","contentSha256":"12f4b34a300f0a7439f5cbf9a29e3fabc778a7fb5ea80631627d1000ab7204f6","textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observedAt":"2026-11-11T23:48:22.265Z"}},{"actor":{"name":"Asur, Sitaram","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Asur, Sitaram","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13183","locator":"head/meta[name=author|citation_author]"},"assertionId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T23:48:22.265Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368","sourceUrl":"https://arxiv.org/abs/2611.13183","contentSha256":"12f4b34a300f0a7439f5cbf9a29e3fabc778a7fb5ea80631627d1000ab7204f6","textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observedAt":"2026-11-11T23:48:22.265Z"}},{"actor":{"name":"Mui, Phil L.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mui, Phil L.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13183","locator":"head/meta[name=author|citation_author]"},"assertionId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-11T23:48:22.265Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368","sourceUrl":"https://arxiv.org/abs/2611.13183","contentSha256":"12f4b34a300f0a7439f5cbf9a29e3fabc778a7fb5ea80631627d1000ab7204f6","textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observedAt":"2026-11-11T23:48:22.265Z"}}],"participationCoverage":{"total":8,"returned":8,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.13183","snapshotId":"40516ababf8ab44afd62f9232be401cae53c5bbe372da200b3a200d8f0a41368","contentSha256":"12f4b34a300f0a7439f5cbf9a29e3fabc778a7fb5ea80631627d1000ab7204f6","textHash":"e4dc6eba60d131d599d529723cf7e9a0a4ce5faced8ca926998a752d2ec0c517","observedAt":"2026-11-11T23:48:22.265Z","basis":"publisher-page-metadata","sourceLinks":47,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/108cfa19bf7f0c7bdf033323b224dcae237ee630ff3b6f11fb0ba1adb215dd6d/source"}}}],"audits":[{"publicationId":"b8b741a4-6be3-4792-810e-db2047bc036d","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The findings and conclusions are consistently attributed to the paper's authors, and the arXiv source is identified correctly. The 'arXiv cs.AI' label matches the supplied publisher and the paper's cross-listing, although its primary category is cs.SE.","excerpt":"Authors:Dipin Khati, Shubham Mehrotra, Bin Bi, Zhou Yu, Denys Poshyvanyk, James Zhu, Sitaram Asur, Phil L. Mui"},{"dimension":"summary","status":"uncertain","explanation":"The summary is faithful, but a general reader cannot follow its central terms. It never explains DSL (domain-specific language) or what AgentScript is: the declarative language Salesforce uses in Agentforce to build conversational agents. The terms pass@3, zero-shot and compilation rate are also unexplained. Smallest repair: spell out DSL, and identify AgentScript as Salesforce's Agentforce agent-definition language. Then gloss pass@3 and zero-shot via tooltips; add them to the term library if missing.","excerpt":"AgentScript, the declarative DSL that Salesforce uses in Agentforce to build conversational agents"},{"dimension":"date","status":"pass","explanation":"The originalPublished value of 2026-11-09 matches the v1 submission date shown on the arXiv page.","excerpt":"[Submitted on 9 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"Every claim and number matches the abstract: 25.0% pass@3, 0% on authoring from scratch, the 31B model's 48.0% compilation rate, roughly doubled accuracy from the loop, and the 28/14/37% skill-pack results. The draft keeps the scope limitation and does not overstate generalization.","excerpt":"Larger models did not reliably outperform smaller ones; what mattered was the harness surrounding the model and the guidance supplied to it."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified, direct link to the original arXiv abstract page.","excerpt":null}],"createdAt":"2026-11-11T23:16:32.390Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}