# Paper tests code agents against semantics-preserving repository rewrites

[Canonical reading](https://escapemeridian.com/readings/paper-tests-code-agents-against-semantics-preserving-repository-b4b7bcc6)

The authors argue that coding agents’ ability to fix software issues can depend on how surrounding code is written, even when a rewrite preserves meaning. They introduce VDL, a probabilistic programming language for defining distributions of repository variants through semantics-preserving transformations (SPTs). This lets evaluators test agents against variations such as routine refactoring or misleading nearby code, rather than relying only on the original repository. In tests across two agentic scaffolds and four models on SWE-bench Verified and SWE-bench Pro, resolve rates fell by up to 6.7 percentage points; the drop was significant in 6 of 16 configurations. The authors also report that model robustness rankings did not hold across scaffolds, describing this as a jagged robustness frontier. As an Escape Meridian interpretation, these results suggest that evaluations on original repositories alone may not capture how agents handle equivalent but differently written code. The abstract describes one generic distribution not targeted at a particular failure mode, alongside examples of what other distributions could reveal. It does not provide enough detail to assess how these results generalize beyond the tested configurations.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv cs.AI · Article

Original source date: Aug 18, 2026 (2026-08-18).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-16T07:01:02.107Z. First observed: 2026-11-16T05:03:52.007Z. Retrieved: 2026-11-16T06:49:44.770Z.

- [Programmable Distributions of Semantics-Preserving Variants for Evaluating Code Agents](https://arxiv.org/abs/2608.18389)

## Limits

The abstract reports results for two agentic scaffolds, four models, and instances from SWE-bench Verified and SWE-bench Pro; it does not establish how the findings generalize beyond those tested configurations. The abstract reports differing robustness rankings across scaffolds but does not explain what accounts for those differences.


## Open question

What accounts for the differences in model robustness rankings across the two agentic scaffolds?


## Source-stated

The authors evaluate whether code agents that repair repository-level issues remain reliable when code surrounding the issue is rewritten into a semantically equivalent form.

> We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form.

- [Supporting source](https://arxiv.org/abs/2608.18389)

## Source-stated

The authors introduce VDL, a probabilistic programming language that expresses distributions of repository variants using semantics-preserving transformations.

> We develop VDL, a probabilistic programming language for expressing such distributions, whose programs randomly apply semantics-preserving transformations (SPTs) to a repository.

- [Supporting source](https://arxiv.org/abs/2608.18389)

## Source-stated

The authors say different variant distributions can reveal sensitivities to routine refactoring or misleading code near a fix.

> Different distributions reveal different aspects of an agent, such as its sensitivity to routine refactoring or to misleading code near the location of a fix.

- [Supporting source](https://arxiv.org/abs/2608.18389)

## Source-stated

In the tested configurations, resolve rates dropped by up to 6.7 percentage points, with significant drops in 6 of 16 configurations.

> Resolve rates drop by up to 6.7 percentage points, significantly in 6 of 16 configurations.

- [Supporting source](https://arxiv.org/abs/2608.18389)

## Source-stated

The authors report that model robustness rankings differed across scaffolds, describing the result as a jagged robustness frontier.

> No ranking of models by robustness holds across scaffolds, revealing a jagged robustness frontier.

- [Supporting source](https://arxiv.org/abs/2608.18389)

## Escape Meridian interpretation

The authors’ results suggest that evaluations on original repositories alone may not capture agent performance on differently written but equivalent code.

> compare the agent's resolve rate on them with that on the original

- [Supporting source](https://arxiv.org/abs/2608.18389)

## Source and contributor context

Mahmud, Hasan Najib, Gupta, Shreya, Chaudhary, Isha, Enis, Nathaniel, Mangal, Ravi, Singh, Gagandeep, Pasareanu, Corina · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Agentic AI

AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.

- [Definition reference](https://developers.google.com/machine-learning/glossary#agentic)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: b96ce4f9-52b0-43f9-8657-9e53055c7fe1.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-16T07:01:02.107Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: d91ab11d2b1fafad1b81c38610364f3839097b1b4cc60d1277bf201d8d76613b. Source text hash: 1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60.


## Revision and source-check history

### Version 1 — 2026-11-16T07:01:02.107Z

The authors argue that coding agents’ ability to fix software issues can depend on how surrounding code is written, even when a rewrite preserves meaning. They introduce VDL, a probabilistic programming language for defining distributions of repository variants through semantics-preserving transformations (SPTs). This lets evaluators test agents against variations such as routine refactoring or misleading nearby code, rather than relying only on the original repository. In tests across two agentic scaffolds and four models on SWE-bench Verified and SWE-bench Pro, resolve rates fell by up to 6.7 percentage points; the drop was significant in 6 of 16 configurations. The authors also report that model robustness rankings did not hold across scaffolds, describing this as a jagged robustness frontier. As an Escape Meridian interpretation, these results suggest that evaluations on original repositories alone may not capture how agents handle equivalent but differently written code. The abstract describes one generic distribution not targeted at a particular failure mode, alongside examples of what other distributions could reveal. It does not provide enough detail to assess how these results generalize beyond the tested configurations.

## Original source

arXiv cs.AI · Article

Original source date: Aug 18, 2026 (2026-08-18).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-16T07:01:02.107Z. First observed: 2026-11-16T05:03:52.007Z. Retrieved: 2026-11-16T06:49:44.770Z.

- [Programmable Distributions of Semantics-Preserving Variants for Evaluating Code Agents](https://arxiv.org/abs/2608.18389)

## Limits

The abstract reports results for two agentic scaffolds, four models, and instances from SWE-bench Verified and SWE-bench Pro; it does not establish how the findings generalize beyond those tested configurations. The abstract reports differing robustness rankings across scaffolds but does not explain what accounts for those differences.


## Open question

What accounts for the differences in model robustness rankings across the two agentic scaffolds?


## Source-stated

The authors evaluate whether code agents that repair repository-level issues remain reliable when code surrounding the issue is rewritten into a semantically equivalent form.

> We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form.

- [Supporting source](https://arxiv.org/abs/2608.18389)

## Source-stated

The authors introduce VDL, a probabilistic programming language that expresses distributions of repository variants using semantics-preserving transformations.

> We develop VDL, a probabilistic programming language for expressing such distributions, whose programs randomly apply semantics-preserving transformations (SPTs) to a repository.

- [Supporting source](https://arxiv.org/abs/2608.18389)

## Source-stated

The authors say different variant distributions can reveal sensitivities to routine refactoring or misleading code near a fix.

> Different distributions reveal different aspects of an agent, such as its sensitivity to routine refactoring or to misleading code near the location of a fix.

- [Supporting source](https://arxiv.org/abs/2608.18389)

## Source-stated

In the tested configurations, resolve rates dropped by up to 6.7 percentage points, with significant drops in 6 of 16 configurations.

> Resolve rates drop by up to 6.7 percentage points, significantly in 6 of 16 configurations.

- [Supporting source](https://arxiv.org/abs/2608.18389)

## Source-stated

The authors report that model robustness rankings differed across scaffolds, describing the result as a jagged robustness frontier.

> No ranking of models by robustness holds across scaffolds, revealing a jagged robustness frontier.

- [Supporting source](https://arxiv.org/abs/2608.18389)

## Escape Meridian interpretation

The authors’ results suggest that evaluations on original repositories alone may not capture agent performance on differently written but equivalent code.

> compare the agent's resolve rate on them with that on the original

- [Supporting source](https://arxiv.org/abs/2608.18389)

## Source and contributor context

Mahmud, Hasan Najib, Gupta, Shreya, Chaudhary, Isha, Enis, Nathaniel, Mangal, Ravi, Singh, Gagandeep, Pasareanu, Corina · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Agentic AI

AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.

- [Definition reference](https://developers.google.com/machine-learning/glossary#agentic)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: b96ce4f9-52b0-43f9-8657-9e53055c7fe1.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-16T07:01:02.107Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: d91ab11d2b1fafad1b81c38610364f3839097b1b4cc60d1277bf201d8d76613b. Source text hash: 1a8ac9c1f8ba436446161f23bfc6fc9cabafc3ebf699e0c267ec6fb85f971c60.


### passed — 2026-11-16T07:01:02.107Z — anthropic/claude-opus-5.5

Publication: b96ce4f9-52b0-43f9-8657-9e53055c7fe1

attribution — pass: The reading attributes the study, VDL, the evaluation design, and the reported results to the paper's authors. Its own inference is explicitly labeled as an Escape Meridian interpretation, and it does not present any source claim as established fact.

> We develop VDL, a probabilistic programming language for expressing such distributions

summary — pass: The title, summary, and question accurately reflect the abstract: semantically equivalent repository rewrites, the VDL language, the evaluation across two scaffolds and four models, drops of up to 6.7 points, and the jagged robustness frontier. The reading says 'preserves meaning,' which explains semantics-preserving well enough for a general reader. Terms such as agentic scaffold can rely on term tooltips. The question follows directly from an unexplained reported result.

> No ranking of models by robustness holds across scaffolds, revealing a jagged robustness frontier.

date — pass: The originalPublished date of 2026-08-18 matches the v1 submission. It is not the v2 revision date of 12 Nov 2026 or the feed date in November.

> \[Submitted on 18 Aug 2026 (v1), last revised 12 Nov 2026 (this version, v2)\]

support — pass: Each source-stated claim matches the abstract. The reading correctly distinguishes the example distributions (refactoring, misleading code) from the single generic distribution actually used in the evaluation. The interpretation that original-only evaluations may miss these effects is conditional, labeled, and grounded in the reported drops. The limitations about generalization and unexplained ranking differences are appropriate and do not overreach.

> For our evaluation, we use it to define a generic distribution that is not targeted at any particular failure mode.

media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most direct access to the source.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/b4b7bcc6c9d581c1cfe61b4a11af30d03bf92f0da0b92450e605be3594a67b50/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/paper-tests-code-agents-against-semantics-preserving-repository-b4b7bcc6.json)
