Zeng, Xiangkai, Liu, Zheng, Lin, Zihao · Nov 4, 2026
Xiangkai Zeng, Zheng Liu and Zihao Lin introduce Multi-step SWE
Markdown · JSON · Original source
Authors Xiangkai Zeng, Zheng Liu and Zihao Lin introduce Multi-step SWE, a designed to test coding agents on chains of software work rather than isolated requests. In its tasks, an agent builds or changes a codebase across 5–15 dependent steps, with each new request starting from code it produced at the previous step. The authors say this setup lets early design decisions and errors carry forward, as they can in continuing development. The benchmark contains 69 tasks across four families of engineering work and nine languages, including from-scratch builds and existing open-source codebases. Each step is grounded in an existing source of truth that defines the instruction and its verifier. Across nine , the strongest model reaches an 18.5% solve rate, and no model solves 38 tasks. The authors say they release the benchmark publicly. The benchmark is intended to address a gap the authors identify in evaluations that score one task at a . Its results do not establish how models perform in real-world software development.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 4, 2026 (2026-11-04).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T01:07:40.569Z. First observed: 2026-11-06T05:10:10.746Z. Retrieved: 2026-11-09T00:55:30.993Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The evidence is the paper's abstract; it does not provide details of individual tasks, evaluation procedures, or reasons for the reported failures. The reported solve rate is for this and does not establish performance in real-world software development.
Open question
What design decisions or errors account for tasks that no model solves across the dependent steps?
Source-stated
The authors introduce a benchmark of software engineering tasks arranged as dependent chains, with each step starting from code produced at the preceding step.
a long-horizon benchmark of 69 software engineering tasks, each a chain of 5-15 dependent steps on a single codebase.
Source-stated
The authors say that early design decisions and errors carry forward in this setup.
so early design decisions and errors carry forward.
Source-stated
The benchmark covers four families of engineering work and nine languages, and includes new builds and existing open-source codebases.
The tasks span four families of engineering work and nine languages, include both from-scratch builds and existing open-source codebases
Source-stated
The authors report that the strongest of nine frontier models reaches an 18.5% solve rate, while 38 tasks are unsolved by every model.
Across nine frontier models, the strongest model reaches a solve rate of only 18.5%, and 38 of the 69 tasks are not solved by any model.
Source-stated
The authors say the benchmark is publicly available.
We release Multi-step SWE publicly at this https URL.
Source and contributor context
Zeng, Xiangkai, Liu, Zheng, Lin, Zihao · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: fb14c9d6-3eb9-45cc-8242-a0bc4765e8df.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-09T01:07:40.569Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 98f1b715fa70756c7e037f1cfdb68d323d0f1bd555a96246c892459966a15a42. Source text hash: b0ae1dc2fac3b5e78a5e8983c341ad86dbca0d5eb4f9093a089a6e1381f88e0f.
Revision and source-check history
Version 1 · 2026-11-09T01:07:40.569Z
Xiangkai Zeng, Zheng Liu and Zihao Lin introduce Multi-step SWE
Authors Xiangkai Zeng, Zheng Liu and Zihao Lin introduce Multi-step SWE, a designed to test coding agents on chains of software work rather than isolated requests. In its tasks, an agent builds or changes a codebase across 5–15 dependent steps, with each new request starting from code it produced at the previous step. The authors say this setup lets early design decisions and errors carry forward, as they can in continuing development. The benchmark contains 69 tasks across four families of engineering work and nine languages, including from-scratch builds and existing open-source codebases. Each step is grounded in an existing source of truth that defines the instruction and its verifier. Across nine , the strongest model reaches an 18.5% solve rate, and no model solves 38 tasks. The authors say they release the benchmark publicly. The benchmark is intended to address a gap the authors identify in evaluations that score one task at a . Its results do not establish how models perform in real-world software development.
Original source
cs.AI · Article
Source details
Original source date: Nov 4, 2026 (2026-11-04).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T01:07:40.569Z. First observed: 2026-11-06T05:10:10.746Z. Retrieved: 2026-11-09T00:55:30.993Z.
Limits
The evidence is the paper's abstract; it does not provide details of individual tasks, evaluation procedures, or reasons for the reported failures. The reported solve rate is for this and does not establish performance in real-world software development.
Open question
What design decisions or errors account for tasks that no model solves across the dependent steps?
Source-stated
The authors introduce a benchmark of software engineering tasks arranged as dependent chains, with each step starting from code produced at the preceding step.
a long-horizon benchmark of 69 software engineering tasks, each a chain of 5-15 dependent steps on a single codebase.
Source-stated
The authors say that early design decisions and errors carry forward in this setup.
so early design decisions and errors carry forward.
Source-stated
The benchmark covers four families of engineering work and nine languages, and includes new builds and existing open-source codebases.
The tasks span four families of engineering work and nine languages, include both from-scratch builds and existing open-source codebases
Source-stated
The authors report that the strongest of nine frontier models reaches an 18.5% solve rate, while 38 tasks are unsolved by every model.
Across nine frontier models, the strongest model reaches a solve rate of only 18.5%, and 38 of the 69 tasks are not solved by any model.
Source-stated
The authors say the benchmark is publicly available.
We release Multi-step SWE publicly at this https URL.
Source and contributor context
Zeng, Xiangkai, Liu, Zheng, Lin, Zihao · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: fb14c9d6-3eb9-45cc-8242-a0bc4765e8df.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-09T01:07:40.569Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 98f1b715fa70756c7e037f1cfdb68d323d0f1bd555a96246c892459966a15a42. Source text hash: b0ae1dc2fac3b5e78a5e8983c341ad86dbca0d5eb4f9093a089a6e1381f88e0f.
passed · 2026-11-09T01:07:40.569Z · anthropic/claude-opus-5.5
Publication: fb14c9d6-3eb9-45cc-8242-a0bc4765e8df
attribution — pass: The paper is correctly attributed to Xiangkai Zeng, Zheng Liu and Zihao Lin. The benchmark design, results and release are framed as the authors' statements, not as established facts.
Authors:Xiangkai Zeng, Zheng Liu, Zihao Lin
summary — pass: The title, summary and question faithfully convey what is new: dependent chains of 5–15 steps in which the agent builds on its own code, compared with single-task scoring. A general reader can follow the actor, the development and the result. Two minor fidelity notes, neither required to pass: the draft drops the authors' 'only' before 18.5%, and it omits that each step is a complete, multi-hour engineering request. Optional smallest repairs: 'reaches a solve rate of only 18.5%' and 'each step is a multi-hour engineering request'. The question is a fair open issue that follows from the unexplained failures.
Every step is a complete, multi-hour engineering request that starts from the code the agent produced in the preceding steps
date — pass: The original publication date of 2026-11-04 matches the v1 arXiv submission date. The 2026-11-06 feed date is correctly not used.
[Submitted on 4 Nov 2026]
support — pass: Every claim matches an exact abstract excerpt. The closing caveat and the limitations stay appropriately bounded: they do not turn the benchmark score into a claim about real-world reliability. The draft invents no benefits, costs or causes for the failures.
Across nine frontier models, the strongest model reaches a solve rate of only 18.5%, and 38 of the 69 tasks are not solved by any model.
media — pass: The selected media is the verified arXiv abstract page, which is the direct original source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference