Deng, Yangshen, Ruan, Jiaqi, Huang, Yeqi, Lau, Nick, Sheng, Guangming, Mai, Luo · Oct 27, 2026
InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?
Markdown · JSON · Original source
Yangshen Deng and coauthors introduce InfraDev, a built from real-world changes to Megatron-LM, vLLM, and verl. Its 36 tasks ask coding agents to reproduce changes spanning many files and run them across many , rather than solve only single-kernel or single-GPU tasks. The authors say hidden probes check requirements that original tests miss, while prompts describe the interfaces without revealing the solution. Across six , the best resolves 38.9% of tasks; hidden probes reject 32% of submissions that pass every official test. With original pull request descriptions as prompts, no model resolves a task. The results suggest that passing a project’s tests can overstate whether an AI infrastructure change works, and that agents’ own tests often miss memory, , or traffic. The authors also report that turning their failure analysis into an agent skill raises one frontier model’s resolved rate on Megatron-LM tasks by 10.5%. How well would these benchmark results predict coding agents’ performance on AI infrastructure changes outside the 36 tasks?
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 27, 2026 (2026-10-27).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-13T22:18:24.416Z. First observed: 2026-10-28T06:06:35.543Z. Retrieved: 2026-10-30T20:32:37.901Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The evidence here is the paper’s abstract, not its full methods or results; it does not provide details needed to assess task selection, evaluation conditions, or the uncertainty around reported rates. The abstract reports results on InfraDev tasks and does not establish how well those results predict performance on other AI infrastructure work.
Open question
How well would InfraDev’s results predict coding agents’ performance on AI infrastructure changes outside its 36 tasks?
Source-stated
The authors introduce InfraDev, a benchmark of 36 tasks based on real-world pull requests to Megatron-LM, vLLM, and verl, covering training, inference, and reinforcement learning infrastructure development.
We introduce InfraDev, a benchmark of 36 tasks built from real-world pull requests to Megatron-LM, vLLM, and verl, covering infrastructure development for training, inference, and reinforcement learning.
Source-stated
InfraDev tasks ask agents to reimplement changes spanning many files and run across many GPUs.
Each task asks an agent to reimplement a real change that spans many files and must run correctly across many GPUs.
Source-stated
The authors report that hidden probes reject some submissions that pass all official tests, and that no model resolves a task when given original pull request descriptions as prompts.
the probes reject 32% of the submissions that pass every official test, and with the original pull request descriptions as prompts, no model resolves a single task.
Source-stated
Across six frontier models, the best resolves 38.9% of InfraDev tasks.
Across six frontier models, the best resolves only 38.9% of the tasks
Source-stated
The authors report that turning their findings into an agent skill raises a frontier model’s resolved rate on Megatron-LM tasks by 10.5%.
Turning these findings into an agent skill raises the resolved rate of a frontier model on the Megatron-LM tasks by 10.5%.
Escape Meridian interpretation
The findings suggest that passing official tests may not establish that a change meets requirements those tests leave unchecked.
the probes reject 32% of the submissions that pass every official test
Source and contributor context
Deng, Yangshen, Ruan, Jiaqi, Huang, Yeqi, Lau, Nick, Sheng, Guangming, Mai, Luo · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Graphics processing unit
A chip that performs many calculations at once; the main hardware used to train and run AI models.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 4611878e-59e0-44c0-8bbf-c7969429b5cf.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-13T22:18:24.416Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 16fb9174289b446e0fc4743d59169958a0b792a8ee44875acebb371deadfc44a. Source text hash: 99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc.
Revision and source-check history
Version 1 · 2026-11-13T22:18:24.416Z
InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?
Yangshen Deng and coauthors introduce InfraDev, a built from real-world changes to Megatron-LM, vLLM, and verl. Its 36 tasks ask coding agents to reproduce changes spanning many files and run them across many , rather than solve only single-kernel or single-GPU tasks. The authors say hidden probes check requirements that original tests miss, while prompts describe the interfaces without revealing the solution. Across six , the best resolves 38.9% of tasks; hidden probes reject 32% of submissions that pass every official test. With original pull request descriptions as prompts, no model resolves a task. The results suggest that passing a project’s tests can overstate whether an AI infrastructure change works, and that agents’ own tests often miss memory, , or traffic. The authors also report that turning their failure analysis into an agent skill raises one frontier model’s resolved rate on Megatron-LM tasks by 10.5%. How well would these benchmark results predict coding agents’ performance on AI infrastructure changes outside the 36 tasks?
Original source
cs.AI · Article
Source details
Original source date: Oct 27, 2026 (2026-10-27).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-13T22:18:24.416Z. First observed: 2026-10-28T06:06:35.543Z. Retrieved: 2026-10-30T20:32:37.901Z.
Limits
The evidence here is the paper’s abstract, not its full methods or results; it does not provide details needed to assess task selection, evaluation conditions, or the uncertainty around reported rates. The abstract reports results on InfraDev tasks and does not establish how well those results predict performance on other AI infrastructure work.
Open question
How well would InfraDev’s results predict coding agents’ performance on AI infrastructure changes outside its 36 tasks?
Source-stated
The authors introduce InfraDev, a benchmark of 36 tasks based on real-world pull requests to Megatron-LM, vLLM, and verl, covering training, inference, and reinforcement learning infrastructure development.
We introduce InfraDev, a benchmark of 36 tasks built from real-world pull requests to Megatron-LM, vLLM, and verl, covering infrastructure development for training, inference, and reinforcement learning.
Source-stated
InfraDev tasks ask agents to reimplement changes spanning many files and run across many GPUs.
Each task asks an agent to reimplement a real change that spans many files and must run correctly across many GPUs.
Source-stated
The authors report that hidden probes reject some submissions that pass all official tests, and that no model resolves a task when given original pull request descriptions as prompts.
the probes reject 32% of the submissions that pass every official test, and with the original pull request descriptions as prompts, no model resolves a single task.
Source-stated
Across six frontier models, the best resolves 38.9% of InfraDev tasks.
Across six frontier models, the best resolves only 38.9% of the tasks
Source-stated
The authors report that turning their findings into an agent skill raises a frontier model’s resolved rate on Megatron-LM tasks by 10.5%.
Turning these findings into an agent skill raises the resolved rate of a frontier model on the Megatron-LM tasks by 10.5%.
Escape Meridian interpretation
The findings suggest that passing official tests may not establish that a change meets requirements those tests leave unchecked.
the probes reject 32% of the submissions that pass every official test
Source and contributor context
Deng, Yangshen, Ruan, Jiaqi, Huang, Yeqi, Lau, Nick, Sheng, Guangming, Mai, Luo · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Graphics processing unit
A chip that performs many calculations at once; the main hardware used to train and run AI models.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 4611878e-59e0-44c0-8bbf-c7969429b5cf.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-13T22:18:24.416Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 16fb9174289b446e0fc4743d59169958a0b792a8ee44875acebb371deadfc44a. Source text hash: 99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc.
uncertain · 2026-11-13T22:18:24.416Z · anthropic/claude-opus-5.5
Publication: 4611878e-59e0-44c0-8bbf-c7969429b5cf
attribution — pass: The reading correctly credits InfraDev to Yangshen Deng and coauthors and attributes the benchmark design, results and agent-skill improvement to the authors.
Authors:Yangshen Deng, Jiaqi Ruan, Yeqi Huang, Nick Lau, Guangming Sheng, Luo Mai
summary — uncertain: The summary is faithful, but a reader without specialist background may not understand it. It never says that Megatron-LM, vLLM and verl are software projects for training, serving and reinforcement-learning large models. It also drops the source's training, inference and RL scope. Terms such as single-kernel, GPUs, hidden probes and agent skill appear without plain explanation or term-library tooltips. The smallest repair is to say what the three projects do and add tooltips for these terms. Smaller issues: the summary repeats the question verbatim, and dropping the source's 'only' before 38.9% slightly softens the authors' framing.
covering infrastructure development for training, inference, and reinforcement learning
date — pass: The originalPublished date of 2026-10-27 matches the v1 arXiv submission date. The 2026-10-29 feed date is later and is correctly not used.
[Submitted on 27 Oct 2026]
support — pass: Every claim matches the abstract: the 36 tasks, multi-file multi-GPU changes, the 38.9% best resolve rate, the 32% of test-passing submissions rejected by probes, zero resolves with the original PR descriptions, and the 10.5% gain from the agent skill. The interpretation that passing tests can overstate correctness is hedged and supported by the abstract's own framing. 'Often miss' fairly paraphrases 'rarely check'. The limitations and the question do not overreach.
grade with tests and prompts that misjudge both correct and incorrect patches
media — pass: The selected media is the verified direct link to the arXiv abstract page for this paper. It is the most direct way for a reader to reach the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv