{"id":"b4b7bcc6c9d581c1cfe61b4a11af30d03bf92f0da0b92450e605be3594a67b50","title":"Paper tests code agents against semantics-preserving repository rewrites","argument":"The authors argue that coding agents’ ability to fix software issues can depend on how surrounding code is written, even when a rewrite preserves meaning. They introduce VDL, a probabilistic programming language for defining distributions of repository variants through semantics-preserving transformations (SPTs). This lets evaluators test agents against variations such as routine refactoring or misleading nearby code, rather than relying only on the original repository. In tests across two agentic scaffolds and four models on SWE-bench Verified and SWE-bench Pro, resolve rates fell by up to 6.7 percentage points; the drop was significant in 6 of 16 configurations. The authors also report that model robustness rankings did not hold across scaffolds, describing this as a jagged robustness frontier. As an Escape Meridian interpretation, these results suggest that evaluations on original repositories alone may not capture how agents handle equivalent but differently written code. The abstract describes one generic distribution not targeted at a particular failure mode, alongside examples of what other distributions could reveal. It does not provide enough detail to assess how these results generalize beyond the tested configurations.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract reports results for two agentic scaffolds, four models, and instances from SWE-bench Verified and SWE-bench Pro; it does not establish how the findings generalize beyond those tested configurations. The abstract reports differing robustness rankings across scaffolds but does not explain what accounts for those differences.","speaker":{"name":"Mahmud, Hasan Najib, Gupta, Shreya, Chaudhary, Isha, Enis, Nathaniel, Mangal, Ravi, Singh, Gagandeep, Pasareanu, Corina","roleAtSource":"Article author"},"source":{"title":"Programmable Distributions of Semantics-Preserving Variants for Evaluating Code Agents","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2608.18389","publishedAt":"2026-08-18","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors evaluate whether code agents that repair repository-level issues remain reliable when code surrounding the issue is rewritten into a semantically equivalent form.","excerpt":"We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form.","url":"https://arxiv.org/abs/2608.18389","relationship":"Source-stated"},{"text":"The authors introduce VDL, a probabilistic programming language that expresses distributions of repository variants using semantics-preserving transformations.","excerpt":"We develop VDL, a probabilistic programming language for expressing such distributions, whose programs randomly apply semantics-preserving transformations (SPTs) to a repository.","url":"https://arxiv.org/abs/2608.18389","relationship":"Source-stated"},{"text":"The authors say different variant distributions can reveal sensitivities to routine refactoring or misleading code near a fix.","excerpt":"Different distributions reveal different aspects of an agent, such as its sensitivity to routine refactoring or to misleading code near the location of a fix.","url":"https://arxiv.org/abs/2608.18389","relationship":"Source-stated"},{"text":"In the tested configurations, resolve rates dropped by up to 6.7 percentage points, with significant drops in 6 of 16 configurations.","excerpt":"Resolve rates drop by up to 6.7 percentage points, significantly in 6 of 16 configurations.","url":"https://arxiv.org/abs/2608.18389","relationship":"Source-stated"},{"text":"The authors report that model robustness rankings differed across scaffolds, describing the result as a jagged robustness frontier.","excerpt":"No ranking of models by robustness holds across scaffolds, revealing a jagged robustness frontier.","url":"https://arxiv.org/abs/2608.18389","relationship":"Source-stated"},{"text":"The authors’ results suggest that evaluations on original repositories alone may not capture agent performance on differently written but equivalent code.","excerpt":"compare the agent's resolve rate on them with that on the original","url":"https://arxiv.org/abs/2608.18389","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2608.18389","checkedAt":"2026-11-16T07:32:09.440Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"What accounts for the differences in model robustness rankings across the two agentic scaffolds?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Paper tests code agents against semantics-preserving repository rewrites","summary":"The authors argue that coding agents’ ability to fix software issues can depend on how surrounding code is written, even when a rewrite preserves meaning. They introduce VDL, a probabilistic programming language for defining distributions of repository variants through semantics-preserving transformations (SPTs). This lets evaluators test agents against variations such as routine refactoring or misleading nearby code, rather than relying only on the original repository. In tests across two agentic scaffolds and four models on SWE-bench Verified and SWE-bench Pro, resolve rates fell by up to 6.7 percentage points; the drop was significant in 6 of 16 configurations. The authors also report that model robustness rankings did not hold across scaffolds, describing this as a jagged robustness frontier. As an Escape Meridian interpretation, these results suggest that evaluations on original repositories alone may not capture how agents handle equivalent but differently written code. The abstract describes one generic distribution not targeted at a particular failure mode, alongside examples of what other distributions could reveal. It does not provide enough detail to assess how these results generalize beyond the tested configurations.","question":"What accounts for the differences in model robustness rankings across the two agentic scaffolds?","claims":[{"text":"The authors evaluate whether code agents that repair repository-level issues remain reliable when code surrounding the issue is rewritten into a semantically equivalent form.","excerpt":"We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form.","support":"source-stated"},{"text":"The authors introduce VDL, a probabilistic programming language that expresses distributions of repository variants using semantics-preserving transformations.","excerpt":"We develop VDL, a probabilistic programming language for expressing such distributions, whose programs randomly apply semantics-preserving transformations (SPTs) to a repository.","support":"source-stated"},{"text":"The authors say different variant distributions can reveal sensitivities to routine refactoring or misleading code near a fix.","excerpt":"Different distributions reveal different aspects of an agent, such as its sensitivity to routine refactoring or to misleading code near the location of a fix.","support":"source-stated"},{"text":"In the tested configurations, resolve rates dropped by up to 6.7 percentage points, with significant drops in 6 of 16 configurations.","excerpt":"Resolve rates drop by up to 6.7 percentage points, significantly in 6 of 16 configurations.","support":"source-stated"},{"text":"The authors report that model robustness rankings differed across scaffolds, describing the result as a jagged robustness frontier.","excerpt":"No ranking of models by robustness holds across scaffolds, revealing a jagged robustness frontier.","support":"source-stated"},{"text":"The authors’ results suggest that evaluations on original repositories alone may not capture agent performance on differently written but equivalent code.","excerpt":"compare the agent's resolve rate on them with that on the original","support":"interpretation"}],"limitations":["The abstract reports results for two agentic scaffolds, four models, and instances from SWE-bench Verified and SWE-bench Pro; it does not establish how the findings generalize beyond those tested configurations.","The abstract reports differing robustness rankings across scaffolds but does not explain what accounts for those differences."],"mediaId":"dc8c63c17f1354e49bc2aaec4f6773aeaa64d671de10eff34bcc43bfd1907f20","correctionReason":null,"id":"b4b7bcc6c9d581c1cfe61b4a11af30d03bf92f0da0b92450e605be3594a67b50","publicationId":"b96ce4f9-52b0-43f9-8657-9e53055c7fe1","version":1,"sourceUrl":"https://arxiv.org/abs/2608.18389","sourceTitle":"Programmable Distributions of Semantics-Preserving Variants for Evaluating Code Agents","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-08-18","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-16T05:03:52.007Z","createdAt":"2026-11-16T07:01:02.107Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"dc8c63c17f1354e49bc2aaec4f6773aeaa64d671de10eff34bcc43bfd1907f20","kind":"article","url":"https://arxiv.org/abs/2608.18389","sourceUrl":"https://arxiv.org/abs/2608.18389","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-16T06:49:45.031Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2608.18389","checkedAt":"2026-11-16T07:32:09.440Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"dc8c63c17f1354e49bc2aaec4f6773aeaa64d671de10eff34bcc43bfd1907f20","confidence":0.7200000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"3085e30e8cb25c6fdd77ed3f0f544b461d32bcaec6108ccbf7350e234063dd05","decidedAt":"2026-11-16T06:49:46.874Z"},"provenance":{"textHash":"1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60","observationId":"d91ab11d2b1fafad1b81c38610364f3839097b1b4cc60d1277bf201d8d76613b","retrievedAt":"2026-11-16T06:49:44.770Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"47b17f77-a1aa-4045-8c3c-abcff2fe4751"},"reviewer":{"model":"gpt-6-luna","runId":"872960a7-5447-4612-8a39-7f55acae6eeb"}},"audit":{"status":"passed","reviewedAt":"2026-11-16T07:01:02.107Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Mahmud, Hasan Najib","url":null,"kind":"unknown"},{"name":"Gupta, Shreya","url":null,"kind":"unknown"},{"name":"Chaudhary, Isha","url":null,"kind":"unknown"},{"name":"Enis, Nathaniel","url":null,"kind":"unknown"},{"name":"Mangal, Ravi","url":null,"kind":"unknown"},{"name":"Singh, Gagandeep","url":null,"kind":"unknown"},{"name":"Pasareanu, Corina","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Mahmud, Hasan Najib","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mahmud, Hasan Najib","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2608.18389","locator":"head/meta[name=author|citation_author]"},"assertionId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-16T07:32:09.440Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9","sourceUrl":"https://arxiv.org/abs/2608.18389","contentSha256":"a1c315f35472093e62881d1afdae6de2e9fa78c6a0247df66d655fc0d346f0f6","textHash":"1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60","observedAt":"2026-11-16T07:32:09.440Z"}},{"actor":{"name":"Gupta, Shreya","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gupta, Shreya","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2608.18389","locator":"head/meta[name=author|citation_author]"},"assertionId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-16T07:32:09.440Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9","sourceUrl":"https://arxiv.org/abs/2608.18389","contentSha256":"a1c315f35472093e62881d1afdae6de2e9fa78c6a0247df66d655fc0d346f0f6","textHash":"1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60","observedAt":"2026-11-16T07:32:09.440Z"}},{"actor":{"name":"Chaudhary, Isha","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chaudhary, Isha","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2608.18389","locator":"head/meta[name=author|citation_author]"},"assertionId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-16T07:32:09.440Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9","sourceUrl":"https://arxiv.org/abs/2608.18389","contentSha256":"a1c315f35472093e62881d1afdae6de2e9fa78c6a0247df66d655fc0d346f0f6","textHash":"1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60","observedAt":"2026-11-16T07:32:09.440Z"}},{"actor":{"name":"Enis, Nathaniel","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Enis, Nathaniel","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2608.18389","locator":"head/meta[name=author|citation_author]"},"assertionId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-16T07:32:09.440Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9","sourceUrl":"https://arxiv.org/abs/2608.18389","contentSha256":"a1c315f35472093e62881d1afdae6de2e9fa78c6a0247df66d655fc0d346f0f6","textHash":"1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60","observedAt":"2026-11-16T07:32:09.440Z"}},{"actor":{"name":"Mangal, Ravi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mangal, Ravi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2608.18389","locator":"head/meta[name=author|citation_author]"},"assertionId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-16T07:32:09.440Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9","sourceUrl":"https://arxiv.org/abs/2608.18389","contentSha256":"a1c315f35472093e62881d1afdae6de2e9fa78c6a0247df66d655fc0d346f0f6","textHash":"1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60","observedAt":"2026-11-16T07:32:09.440Z"}},{"actor":{"name":"Singh, Gagandeep","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Singh, Gagandeep","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2608.18389","locator":"head/meta[name=author|citation_author]"},"assertionId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-16T07:32:09.440Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9","sourceUrl":"https://arxiv.org/abs/2608.18389","contentSha256":"a1c315f35472093e62881d1afdae6de2e9fa78c6a0247df66d655fc0d346f0f6","textHash":"1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60","observedAt":"2026-11-16T07:32:09.440Z"}},{"actor":{"name":"Pasareanu, Corina","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Pasareanu, Corina","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2608.18389","locator":"head/meta[name=author|citation_author]"},"assertionId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-16T07:32:09.440Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9","sourceUrl":"https://arxiv.org/abs/2608.18389","contentSha256":"a1c315f35472093e62881d1afdae6de2e9fa78c6a0247df66d655fc0d346f0f6","textHash":"1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60","observedAt":"2026-11-16T07:32:09.440Z"}}],"participationCoverage":{"total":7,"returned":7,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2608.18389","snapshotId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9","contentSha256":"a1c315f35472093e62881d1afdae6de2e9fa78c6a0247df66d655fc0d346f0f6","textHash":"1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60","observedAt":"2026-11-16T07:32:09.440Z","basis":"publisher-page-metadata","sourceLinks":48,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/b4b7bcc6c9d581c1cfe61b4a11af30d03bf92f0da0b92450e605be3594a67b50/source"}},"canonicalUrl":"https://escapemeridian.com/readings/paper-tests-code-agents-against-semantics-preserving-repository-b4b7bcc6","formats":{"html":"/readings/paper-tests-code-agents-against-semantics-preserving-repository-b4b7bcc6","markdown":"/readings/paper-tests-code-agents-against-semantics-preserving-repository-b4b7bcc6.md","json":"/readings/paper-tests-code-agents-against-semantics-preserving-repository-b4b7bcc6.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-agentic","name":"Agentic AI","explanation":"AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.","referenceUrl":"https://developers.google.com/machine-learning/glossary#agentic"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"b4b7bcc6c9d581c1cfe61b4a11af30d03bf92f0da0b92450e605be3594a67b50","revision":"b96ce4f9-52b0-43f9-8657-9e53055c7fe1","revisionKind":"publication-id","title":"Paper tests code agents against semantics-preserving repository rewrites","url":"https://escapemeridian.com/readings/paper-tests-code-agents-against-semantics-preserving-repository-b4b7bcc6"},"source":{"type":"source","id":"https://arxiv.org/abs/2608.18389","revision":"1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60","revisionKind":"sha256-source-text","title":"Programmable Distributions of Semantics-Preserving Variants for Evaluating Code Agents","url":"https://arxiv.org/abs/2608.18389"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"b4b7bcc6c9d581c1cfe61b4a11af30d03bf92f0da0b92450e605be3594a67b50","title":"Paper tests code agents against semantics-preserving repository rewrites","argument":"The authors argue that coding agents’ ability to fix software issues can depend on how surrounding code is written, even when a rewrite preserves meaning. They introduce VDL, a probabilistic programming language for defining distributions of repository variants through semantics-preserving transformations (SPTs). This lets evaluators test agents against variations such as routine refactoring or misleading nearby code, rather than relying only on the original repository. In tests across two agentic scaffolds and four models on SWE-bench Verified and SWE-bench Pro, resolve rates fell by up to 6.7 percentage points; the drop was significant in 6 of 16 configurations. The authors also report that model robustness rankings did not hold across scaffolds, describing this as a jagged robustness frontier. As an Escape Meridian interpretation, these results suggest that evaluations on original repositories alone may not capture how agents handle equivalent but differently written code. The abstract describes one generic distribution not targeted at a particular failure mode, alongside examples of what other distributions could reveal. It does not provide enough detail to assess how these results generalize beyond the tested configurations.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract reports results for two agentic scaffolds, four models, and instances from SWE-bench Verified and SWE-bench Pro; it does not establish how the findings generalize beyond those tested configurations. The abstract reports differing robustness rankings across scaffolds but does not explain what accounts for those differences.","speaker":{"name":"Mahmud, Hasan Najib, Gupta, Shreya, Chaudhary, Isha, Enis, Nathaniel, Mangal, Ravi, Singh, Gagandeep, Pasareanu, Corina","roleAtSource":"Article author"},"source":{"title":"Programmable Distributions of Semantics-Preserving Variants for Evaluating Code Agents","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2608.18389","publishedAt":"2026-08-18","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors evaluate whether code agents that repair repository-level issues remain reliable when code surrounding the issue is rewritten into a semantically equivalent form.","excerpt":"We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form.","url":"https://arxiv.org/abs/2608.18389","relationship":"Source-stated"},{"text":"The authors introduce VDL, a probabilistic programming language that expresses distributions of repository variants using semantics-preserving transformations.","excerpt":"We develop VDL, a probabilistic programming language for expressing such distributions, whose programs randomly apply semantics-preserving transformations (SPTs) to a repository.","url":"https://arxiv.org/abs/2608.18389","relationship":"Source-stated"},{"text":"The authors say different variant distributions can reveal sensitivities to routine refactoring or misleading code near a fix.","excerpt":"Different distributions reveal different aspects of an agent, such as its sensitivity to routine refactoring or to misleading code near the location of a fix.","url":"https://arxiv.org/abs/2608.18389","relationship":"Source-stated"},{"text":"In the tested configurations, resolve rates dropped by up to 6.7 percentage points, with significant drops in 6 of 16 configurations.","excerpt":"Resolve rates drop by up to 6.7 percentage points, significantly in 6 of 16 configurations.","url":"https://arxiv.org/abs/2608.18389","relationship":"Source-stated"},{"text":"The authors report that model robustness rankings differed across scaffolds, describing the result as a jagged robustness frontier.","excerpt":"No ranking of models by robustness holds across scaffolds, revealing a jagged robustness frontier.","url":"https://arxiv.org/abs/2608.18389","relationship":"Source-stated"},{"text":"The authors’ results suggest that evaluations on original repositories alone may not capture agent performance on differently written but equivalent code.","excerpt":"compare the agent's resolve rate on them with that on the original","url":"https://arxiv.org/abs/2608.18389","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2608.18389","checkedAt":"2026-11-16T07:32:09.440Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"What accounts for the differences in model robustness rankings across the two agentic scaffolds?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Paper tests code agents against semantics-preserving repository rewrites","summary":"The authors argue that coding agents’ ability to fix software issues can depend on how surrounding code is written, even when a rewrite preserves meaning. They introduce VDL, a probabilistic programming language for defining distributions of repository variants through semantics-preserving transformations (SPTs). This lets evaluators test agents against variations such as routine refactoring or misleading nearby code, rather than relying only on the original repository. In tests across two agentic scaffolds and four models on SWE-bench Verified and SWE-bench Pro, resolve rates fell by up to 6.7 percentage points; the drop was significant in 6 of 16 configurations. The authors also report that model robustness rankings did not hold across scaffolds, describing this as a jagged robustness frontier. As an Escape Meridian interpretation, these results suggest that evaluations on original repositories alone may not capture how agents handle equivalent but differently written code. The abstract describes one generic distribution not targeted at a particular failure mode, alongside examples of what other distributions could reveal. It does not provide enough detail to assess how these results generalize beyond the tested configurations.","question":"What accounts for the differences in model robustness rankings across the two agentic scaffolds?","claims":[{"text":"The authors evaluate whether code agents that repair repository-level issues remain reliable when code surrounding the issue is rewritten into a semantically equivalent form.","excerpt":"We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form.","support":"source-stated"},{"text":"The authors introduce VDL, a probabilistic programming language that expresses distributions of repository variants using semantics-preserving transformations.","excerpt":"We develop VDL, a probabilistic programming language for expressing such distributions, whose programs randomly apply semantics-preserving transformations (SPTs) to a repository.","support":"source-stated"},{"text":"The authors say different variant distributions can reveal sensitivities to routine refactoring or misleading code near a fix.","excerpt":"Different distributions reveal different aspects of an agent, such as its sensitivity to routine refactoring or to misleading code near the location of a fix.","support":"source-stated"},{"text":"In the tested configurations, resolve rates dropped by up to 6.7 percentage points, with significant drops in 6 of 16 configurations.","excerpt":"Resolve rates drop by up to 6.7 percentage points, significantly in 6 of 16 configurations.","support":"source-stated"},{"text":"The authors report that model robustness rankings differed across scaffolds, describing the result as a jagged robustness frontier.","excerpt":"No ranking of models by robustness holds across scaffolds, revealing a jagged robustness frontier.","support":"source-stated"},{"text":"The authors’ results suggest that evaluations on original repositories alone may not capture agent performance on differently written but equivalent code.","excerpt":"compare the agent's resolve rate on them with that on the original","support":"interpretation"}],"limitations":["The abstract reports results for two agentic scaffolds, four models, and instances from SWE-bench Verified and SWE-bench Pro; it does not establish how the findings generalize beyond those tested configurations.","The abstract reports differing robustness rankings across scaffolds but does not explain what accounts for those differences."],"mediaId":"dc8c63c17f1354e49bc2aaec4f6773aeaa64d671de10eff34bcc43bfd1907f20","correctionReason":null,"id":"b4b7bcc6c9d581c1cfe61b4a11af30d03bf92f0da0b92450e605be3594a67b50","publicationId":"b96ce4f9-52b0-43f9-8657-9e53055c7fe1","version":1,"sourceUrl":"https://arxiv.org/abs/2608.18389","sourceTitle":"Programmable Distributions of Semantics-Preserving Variants for Evaluating Code Agents","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-08-18","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-16T05:03:52.007Z","createdAt":"2026-11-16T07:01:02.107Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"dc8c63c17f1354e49bc2aaec4f6773aeaa64d671de10eff34bcc43bfd1907f20","kind":"article","url":"https://arxiv.org/abs/2608.18389","sourceUrl":"https://arxiv.org/abs/2608.18389","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-16T06:49:45.031Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2608.18389","checkedAt":"2026-11-16T07:32:09.440Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"dc8c63c17f1354e49bc2aaec4f6773aeaa64d671de10eff34bcc43bfd1907f20","confidence":0.7200000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"3085e30e8cb25c6fdd77ed3f0f544b461d32bcaec6108ccbf7350e234063dd05","decidedAt":"2026-11-16T06:49:46.874Z"},"provenance":{"textHash":"1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60","observationId":"d91ab11d2b1fafad1b81c38610364f3839097b1b4cc60d1277bf201d8d76613b","retrievedAt":"2026-11-16T06:49:44.770Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"47b17f77-a1aa-4045-8c3c-abcff2fe4751"},"reviewer":{"model":"gpt-6-luna","runId":"872960a7-5447-4612-8a39-7f55acae6eeb"}},"audit":{"status":"passed","reviewedAt":"2026-11-16T07:01:02.107Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Mahmud, Hasan Najib","url":null,"kind":"unknown"},{"name":"Gupta, Shreya","url":null,"kind":"unknown"},{"name":"Chaudhary, Isha","url":null,"kind":"unknown"},{"name":"Enis, Nathaniel","url":null,"kind":"unknown"},{"name":"Mangal, Ravi","url":null,"kind":"unknown"},{"name":"Singh, Gagandeep","url":null,"kind":"unknown"},{"name":"Pasareanu, Corina","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Mahmud, Hasan Najib","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mahmud, Hasan Najib","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2608.18389","locator":"head/meta[name=author|citation_author]"},"assertionId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-16T07:32:09.440Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9","sourceUrl":"https://arxiv.org/abs/2608.18389","contentSha256":"a1c315f35472093e62881d1afdae6de2e9fa78c6a0247df66d655fc0d346f0f6","textHash":"1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60","observedAt":"2026-11-16T07:32:09.440Z"}},{"actor":{"name":"Gupta, Shreya","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gupta, Shreya","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2608.18389","locator":"head/meta[name=author|citation_author]"},"assertionId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-16T07:32:09.440Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9","sourceUrl":"https://arxiv.org/abs/2608.18389","contentSha256":"a1c315f35472093e62881d1afdae6de2e9fa78c6a0247df66d655fc0d346f0f6","textHash":"1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60","observedAt":"2026-11-16T07:32:09.440Z"}},{"actor":{"name":"Chaudhary, Isha","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chaudhary, Isha","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2608.18389","locator":"head/meta[name=author|citation_author]"},"assertionId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-16T07:32:09.440Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9","sourceUrl":"https://arxiv.org/abs/2608.18389","contentSha256":"a1c315f35472093e62881d1afdae6de2e9fa78c6a0247df66d655fc0d346f0f6","textHash":"1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60","observedAt":"2026-11-16T07:32:09.440Z"}},{"actor":{"name":"Enis, Nathaniel","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Enis, Nathaniel","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2608.18389","locator":"head/meta[name=author|citation_author]"},"assertionId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-16T07:32:09.440Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9","sourceUrl":"https://arxiv.org/abs/2608.18389","contentSha256":"a1c315f35472093e62881d1afdae6de2e9fa78c6a0247df66d655fc0d346f0f6","textHash":"1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60","observedAt":"2026-11-16T07:32:09.440Z"}},{"actor":{"name":"Mangal, Ravi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mangal, Ravi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2608.18389","locator":"head/meta[name=author|citation_author]"},"assertionId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-16T07:32:09.440Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9","sourceUrl":"https://arxiv.org/abs/2608.18389","contentSha256":"a1c315f35472093e62881d1afdae6de2e9fa78c6a0247df66d655fc0d346f0f6","textHash":"1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60","observedAt":"2026-11-16T07:32:09.440Z"}},{"actor":{"name":"Singh, Gagandeep","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Singh, Gagandeep","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2608.18389","locator":"head/meta[name=author|citation_author]"},"assertionId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-16T07:32:09.440Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9","sourceUrl":"https://arxiv.org/abs/2608.18389","contentSha256":"a1c315f35472093e62881d1afdae6de2e9fa78c6a0247df66d655fc0d346f0f6","textHash":"1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60","observedAt":"2026-11-16T07:32:09.440Z"}},{"actor":{"name":"Pasareanu, Corina","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Pasareanu, Corina","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2608.18389","locator":"head/meta[name=author|citation_author]"},"assertionId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-16T07:32:09.440Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9","sourceUrl":"https://arxiv.org/abs/2608.18389","contentSha256":"a1c315f35472093e62881d1afdae6de2e9fa78c6a0247df66d655fc0d346f0f6","textHash":"1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60","observedAt":"2026-11-16T07:32:09.440Z"}}],"participationCoverage":{"total":7,"returned":7,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2608.18389","snapshotId":"763b948f317ab26c4f3b71c87a5eaeb19883bde110019ca6cbc7680f691293b9","contentSha256":"a1c315f35472093e62881d1afdae6de2e9fa78c6a0247df66d655fc0d346f0f6","textHash":"1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60","observedAt":"2026-11-16T07:32:09.440Z","basis":"publisher-page-metadata","sourceLinks":48,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/b4b7bcc6c9d581c1cfe61b4a11af30d03bf92f0da0b92450e605be3594a67b50/source"}}}],"audits":[{"publicationId":"b96ce4f9-52b0-43f9-8657-9e53055c7fe1","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the study, VDL, the evaluation design, and the reported results to the paper's authors. Its own inference is explicitly labeled as an Escape Meridian interpretation, and it does not present any source claim as established fact.","excerpt":"We develop VDL, a probabilistic programming language for expressing such distributions"},{"dimension":"summary","status":"pass","explanation":"The title, summary, and question accurately reflect the abstract: semantically equivalent repository rewrites, the VDL language, the evaluation across two scaffolds and four models, drops of up to 6.7 points, and the jagged robustness frontier. The reading says 'preserves meaning,' which explains semantics-preserving well enough for a general reader. Terms such as agentic scaffold can rely on term tooltips. The question follows directly from an unexplained reported result.","excerpt":"No ranking of models by robustness holds across scaffolds, revealing a jagged robustness frontier."},{"dimension":"date","status":"pass","explanation":"The originalPublished date of 2026-08-18 matches the v1 submission. It is not the v2 revision date of 12 Nov 2026 or the feed date in November.","excerpt":"[Submitted on 18 Aug 2026 (v1), last revised 12 Nov 2026 (this version, v2)]"},{"dimension":"support","status":"pass","explanation":"Each source-stated claim matches the abstract. The reading correctly distinguishes the example distributions (refactoring, misleading code) from the single generic distribution actually used in the evaluation. The interpretation that original-only evaluations may miss these effects is conditional, labeled, and grounded in the reported drops. The limitations about generalization and unexplained ranking differences are appropriate and do not overreach.","excerpt":"For our evaluation, we use it to define a generic distribution that is not targeted at any particular failure mode."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct arXiv abstract page for this paper. It is the most direct access to the source.","excerpt":null}],"createdAt":"2026-11-16T07:01:02.107Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}