# InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell? [Canonical reading](https://escapemeridian.com/readings/infradev-can-coding-agents-develop-ai-infrastructure-and-how-bcb48753) Yangshen Deng and coauthors introduce InfraDev, a benchmark built from real-world changes to Megatron-LM, vLLM, and verl. Its 36 tasks ask coding agents to reproduce changes spanning many files and run them across many GPUs, rather than solve only single-kernel or single-GPU tasks. The authors say hidden probes check requirements that original tests miss, while prompts describe the interfaces without revealing the solution. Across six frontier models, the best resolves 38.9% of tasks; hidden probes reject 32% of submissions that pass every official test. With original pull request descriptions as prompts, no model resolves a task. The results suggest that passing a project’s tests can overstate whether an AI infrastructure change works, and that agents’ own tests often miss memory, time, or traffic. The authors also report that turning their failure analysis into an agent skill raises one frontier model’s resolved rate on Megatron-LM tasks by 10.5%. How well would these benchmark results predict coding agents’ performance on AI infrastructure changes outside the 36 tasks? Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Oct 27, 2026 (2026-10-27). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-13T22:18:24.416Z. First observed: 2026-10-28T06:06:35.543Z. Retrieved: 2026-10-30T20:32:37.901Z. - [InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?](https://arxiv.org/abs/2610.36589) ## Limits The evidence here is the paper’s abstract, not its full methods or results; it does not provide details needed to assess task selection, evaluation conditions, or the uncertainty around reported rates. The abstract reports results on InfraDev tasks and does not establish how well those results predict performance on other AI infrastructure work. ## Open question How well would InfraDev’s results predict coding agents’ performance on AI infrastructure changes outside its 36 tasks? ## Source-stated The authors introduce InfraDev, a benchmark of 36 tasks based on real-world pull requests to Megatron-LM, vLLM, and verl, covering training, inference, and reinforcement learning infrastructure development. > We introduce InfraDev, a benchmark of 36 tasks built from real-world pull requests to Megatron-LM, vLLM, and verl, covering infrastructure development for training, inference, and reinforcement learning. - [Supporting source](https://arxiv.org/abs/2610.36589) ## Source-stated InfraDev tasks ask agents to reimplement changes spanning many files and run across many GPUs. > Each task asks an agent to reimplement a real change that spans many files and must run correctly across many GPUs. - [Supporting source](https://arxiv.org/abs/2610.36589) ## Source-stated The authors report that hidden probes reject some submissions that pass all official tests, and that no model resolves a task when given original pull request descriptions as prompts. > the probes reject 32% of the submissions that pass every official test, and with the original pull request descriptions as prompts, no model resolves a single task. - [Supporting source](https://arxiv.org/abs/2610.36589) ## Source-stated Across six frontier models, the best resolves 38.9% of InfraDev tasks. > Across six frontier models, the best resolves only 38.9% of the tasks - [Supporting source](https://arxiv.org/abs/2610.36589) ## Source-stated The authors report that turning their findings into an agent skill raises a frontier model’s resolved rate on Megatron-LM tasks by 10.5%. > Turning these findings into an agent skill raises the resolved rate of a frontier model on the Megatron-LM tasks by 10.5%. - [Supporting source](https://arxiv.org/abs/2610.36589) ## Escape Meridian interpretation The findings suggest that passing official tests may not establish that a change meets requirements those tests leave unchecked. > the probes reject 32% of the submissions that pass every official test - [Supporting source](https://arxiv.org/abs/2610.36589) ## Source and contributor context Deng, Yangshen, Ruan, Jiaqi, Huang, Yeqi, Lau, Nick, Sheng, Guangming, Mai, Luo · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Graphics processing unit A chip that performs many calculations at once; the main hardware used to train and run AI models. - [Definition reference](https://en.wikipedia.org/wiki/Graphics_processing_unit) ## Frontier model One of the most capable AI models available at a given time; the set shifts as models improve. - [Definition reference](https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## TIME TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services. - [Explore this context](/entities/time) - [Reference](https://time.com/) ## Editorial provenance Version 1. Publication ID: 4611878e-59e0-44c0-8bbf-c7969429b5cf. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-13T22:18:24.416Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 16fb9174289b446e0fc4743d59169958a0b792a8ee44875acebb371deadfc44a. Source text hash: 99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc. ## Revision and source-check history ### Version 1 — 2026-11-13T22:18:24.416Z Yangshen Deng and coauthors introduce InfraDev, a benchmark built from real-world changes to Megatron-LM, vLLM, and verl. Its 36 tasks ask coding agents to reproduce changes spanning many files and run them across many GPUs, rather than solve only single-kernel or single-GPU tasks. The authors say hidden probes check requirements that original tests miss, while prompts describe the interfaces without revealing the solution. Across six frontier models, the best resolves 38.9% of tasks; hidden probes reject 32% of submissions that pass every official test. With original pull request descriptions as prompts, no model resolves a task. The results suggest that passing a project’s tests can overstate whether an AI infrastructure change works, and that agents’ own tests often miss memory, time, or traffic. The authors also report that turning their failure analysis into an agent skill raises one frontier model’s resolved rate on Megatron-LM tasks by 10.5%. How well would these benchmark results predict coding agents’ performance on AI infrastructure changes outside the 36 tasks? ## Original source arXiv cs.AI · Article Original source date: Oct 27, 2026 (2026-10-27). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-13T22:18:24.416Z. First observed: 2026-10-28T06:06:35.543Z. Retrieved: 2026-10-30T20:32:37.901Z. - [InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?](https://arxiv.org/abs/2610.36589) ## Limits The evidence here is the paper’s abstract, not its full methods or results; it does not provide details needed to assess task selection, evaluation conditions, or the uncertainty around reported rates. The abstract reports results on InfraDev tasks and does not establish how well those results predict performance on other AI infrastructure work. ## Open question How well would InfraDev’s results predict coding agents’ performance on AI infrastructure changes outside its 36 tasks? ## Source-stated The authors introduce InfraDev, a benchmark of 36 tasks based on real-world pull requests to Megatron-LM, vLLM, and verl, covering training, inference, and reinforcement learning infrastructure development. > We introduce InfraDev, a benchmark of 36 tasks built from real-world pull requests to Megatron-LM, vLLM, and verl, covering infrastructure development for training, inference, and reinforcement learning. - [Supporting source](https://arxiv.org/abs/2610.36589) ## Source-stated InfraDev tasks ask agents to reimplement changes spanning many files and run across many GPUs. > Each task asks an agent to reimplement a real change that spans many files and must run correctly across many GPUs. - [Supporting source](https://arxiv.org/abs/2610.36589) ## Source-stated The authors report that hidden probes reject some submissions that pass all official tests, and that no model resolves a task when given original pull request descriptions as prompts. > the probes reject 32% of the submissions that pass every official test, and with the original pull request descriptions as prompts, no model resolves a single task. - [Supporting source](https://arxiv.org/abs/2610.36589) ## Source-stated Across six frontier models, the best resolves 38.9% of InfraDev tasks. > Across six frontier models, the best resolves only 38.9% of the tasks - [Supporting source](https://arxiv.org/abs/2610.36589) ## Source-stated The authors report that turning their findings into an agent skill raises a frontier model’s resolved rate on Megatron-LM tasks by 10.5%. > Turning these findings into an agent skill raises the resolved rate of a frontier model on the Megatron-LM tasks by 10.5%. - [Supporting source](https://arxiv.org/abs/2610.36589) ## Escape Meridian interpretation The findings suggest that passing official tests may not establish that a change meets requirements those tests leave unchecked. > the probes reject 32% of the submissions that pass every official test - [Supporting source](https://arxiv.org/abs/2610.36589) ## Source and contributor context Deng, Yangshen, Ruan, Jiaqi, Huang, Yeqi, Lau, Nick, Sheng, Guangming, Mai, Luo · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Graphics processing unit A chip that performs many calculations at once; the main hardware used to train and run AI models. - [Definition reference](https://en.wikipedia.org/wiki/Graphics_processing_unit) ## Frontier model One of the most capable AI models available at a given time; the set shifts as models improve. - [Definition reference](https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## TIME TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services. - [Explore this context](/entities/time) - [Reference](https://time.com/) ## Editorial provenance Version 1. Publication ID: 4611878e-59e0-44c0-8bbf-c7969429b5cf. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-13T22:18:24.416Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 16fb9174289b446e0fc4743d59169958a0b792a8ee44875acebb371deadfc44a. Source text hash: 99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc. ### uncertain — 2026-11-13T22:18:24.416Z — anthropic/claude-opus-5.5 Publication: 4611878e-59e0-44c0-8bbf-c7969429b5cf attribution — pass: The reading correctly credits InfraDev to Yangshen Deng and coauthors and attributes the benchmark design, results and agent-skill improvement to the authors. > Authors:Yangshen Deng, Jiaqi Ruan, Yeqi Huang, Nick Lau, Guangming Sheng, Luo Mai summary — uncertain: The summary is faithful, but a reader without specialist background may not understand it. It never says that Megatron-LM, vLLM and verl are software projects for training, serving and reinforcement-learning large models. It also drops the source's training, inference and RL scope. Terms such as single-kernel, GPUs, hidden probes and agent skill appear without plain explanation or term-library tooltips. The smallest repair is to say what the three projects do and add tooltips for these terms. Smaller issues: the summary repeats the question verbatim, and dropping the source's 'only' before 38.9% slightly softens the authors' framing. > covering infrastructure development for training, inference, and reinforcement learning date — pass: The originalPublished date of 2026-10-27 matches the v1 arXiv submission date. The 2026-10-29 feed date is later and is correctly not used. > \[Submitted on 27 Oct 2026\] support — pass: Every claim matches the abstract: the 36 tasks, multi-file multi-GPU changes, the 38.9% best resolve rate, the 32% of test-passing submissions rejected by probes, zero resolves with the original PR descriptions, and the 10.5% gain from the agent skill. The interpretation that passing tests can overstate correctness is hedged and supported by the abstract's own framing. 'Often miss' fairly paraphrases 'rarely check'. The limitations and the question do not overreach. > grade with tests and prompts that misjudge both correct and incorrect patches media — pass: The selected media is the verified direct link to the arXiv abstract page for this paper. It is the most direct way for a reader to reach the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/bcb4875389a27651f30707b545fd7f466e60386ede3e3ae8ca8e8ca82b0d8527/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/infradev-can-coding-agents-develop-ai-infrastructure-and-how-bcb48753.json)