{"id":"bcb4875389a27651f30707b545fd7f466e60386ede3e3ae8ca8e8ca82b0d8527","title":"InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?","argument":"Yangshen Deng and coauthors introduce InfraDev, a benchmark built from real-world changes to Megatron-LM, vLLM, and verl. Its 36 tasks ask coding agents to reproduce changes spanning many files and run them across many GPUs, rather than solve only single-kernel or single-GPU tasks. The authors say hidden probes check requirements that original tests miss, while prompts describe the interfaces without revealing the solution. Across six frontier models, the best resolves 38.9% of tasks; hidden probes reject 32% of submissions that pass every official test. With original pull request descriptions as prompts, no model resolves a task. The results suggest that passing a project’s tests can overstate whether an AI infrastructure change works, and that agents’ own tests often miss memory, time, or traffic. The authors also report that turning their failure analysis into an agent skill raises one frontier model’s resolved rate on Megatron-LM tasks by 10.5%. How well would these benchmark results predict coding agents’ performance on AI infrastructure changes outside the 36 tasks?","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence here is the paper’s abstract, not its full methods or results; it does not provide details needed to assess task selection, evaluation conditions, or the uncertainty around reported rates. The abstract reports results on InfraDev tasks and does not establish how well those results predict performance on other AI infrastructure work.","speaker":{"name":"Deng, Yangshen, Ruan, Jiaqi, Huang, Yeqi, Lau, Nick, Sheng, Guangming, Mai, Luo","roleAtSource":"Article author"},"source":{"title":"InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.36589","publishedAt":"2026-10-27","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce InfraDev, a benchmark of 36 tasks based on real-world pull requests to Megatron-LM, vLLM, and verl, covering training, inference, and reinforcement learning infrastructure development.","excerpt":"We introduce InfraDev, a benchmark of 36 tasks built from real-world pull requests to Megatron-LM, vLLM, and verl, covering infrastructure development for training, inference, and reinforcement learning.","url":"https://arxiv.org/abs/2610.36589","relationship":"Source-stated"},{"text":"InfraDev tasks ask agents to reimplement changes spanning many files and run across many GPUs.","excerpt":"Each task asks an agent to reimplement a real change that spans many files and must run correctly across many GPUs.","url":"https://arxiv.org/abs/2610.36589","relationship":"Source-stated"},{"text":"The authors report that hidden probes reject some submissions that pass all official tests, and that no model resolves a task when given original pull request descriptions as prompts.","excerpt":"the probes reject 32% of the submissions that pass every official test, and with the original pull request descriptions as prompts, no model resolves a single task.","url":"https://arxiv.org/abs/2610.36589","relationship":"Source-stated"},{"text":"Across six frontier models, the best resolves 38.9% of InfraDev tasks.","excerpt":"Across six frontier models, the best resolves only 38.9% of the tasks","url":"https://arxiv.org/abs/2610.36589","relationship":"Source-stated"},{"text":"The authors report that turning their findings into an agent skill raises a frontier model’s resolved rate on Megatron-LM tasks by 10.5%.","excerpt":"Turning these findings into an agent skill raises the resolved rate of a frontier model on the Megatron-LM tasks by 10.5%.","url":"https://arxiv.org/abs/2610.36589","relationship":"Source-stated"},{"text":"The findings suggest that passing official tests may not establish that a change meets requirements those tests leave unchecked.","excerpt":"the probes reject 32% of the submissions that pass every official test","url":"https://arxiv.org/abs/2610.36589","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.36589","checkedAt":"2026-11-14T00:26:07.746Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well would InfraDev’s results predict coding agents’ performance on AI infrastructure changes outside its 36 tasks?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?","summary":"Yangshen Deng and coauthors introduce InfraDev, a benchmark built from real-world changes to Megatron-LM, vLLM, and verl. Its 36 tasks ask coding agents to reproduce changes spanning many files and run them across many GPUs, rather than solve only single-kernel or single-GPU tasks. The authors say hidden probes check requirements that original tests miss, while prompts describe the interfaces without revealing the solution. Across six frontier models, the best resolves 38.9% of tasks; hidden probes reject 32% of submissions that pass every official test. With original pull request descriptions as prompts, no model resolves a task. The results suggest that passing a project’s tests can overstate whether an AI infrastructure change works, and that agents’ own tests often miss memory, time, or traffic. The authors also report that turning their failure analysis into an agent skill raises one frontier model’s resolved rate on Megatron-LM tasks by 10.5%. How well would these benchmark results predict coding agents’ performance on AI infrastructure changes outside the 36 tasks?","question":"How well would InfraDev’s results predict coding agents’ performance on AI infrastructure changes outside its 36 tasks?","claims":[{"text":"The authors introduce InfraDev, a benchmark of 36 tasks based on real-world pull requests to Megatron-LM, vLLM, and verl, covering training, inference, and reinforcement learning infrastructure development.","excerpt":"We introduce InfraDev, a benchmark of 36 tasks built from real-world pull requests to Megatron-LM, vLLM, and verl, covering infrastructure development for training, inference, and reinforcement learning.","support":"source-stated"},{"text":"InfraDev tasks ask agents to reimplement changes spanning many files and run across many GPUs.","excerpt":"Each task asks an agent to reimplement a real change that spans many files and must run correctly across many GPUs.","support":"source-stated"},{"text":"The authors report that hidden probes reject some submissions that pass all official tests, and that no model resolves a task when given original pull request descriptions as prompts.","excerpt":"the probes reject 32% of the submissions that pass every official test, and with the original pull request descriptions as prompts, no model resolves a single task.","support":"source-stated"},{"text":"Across six frontier models, the best resolves 38.9% of InfraDev tasks.","excerpt":"Across six frontier models, the best resolves only 38.9% of the tasks","support":"source-stated"},{"text":"The authors report that turning their findings into an agent skill raises a frontier model’s resolved rate on Megatron-LM tasks by 10.5%.","excerpt":"Turning these findings into an agent skill raises the resolved rate of a frontier model on the Megatron-LM tasks by 10.5%.","support":"source-stated"},{"text":"The findings suggest that passing official tests may not establish that a change meets requirements those tests leave unchecked.","excerpt":"the probes reject 32% of the submissions that pass every official test","support":"interpretation"}],"limitations":["The evidence here is the paper’s abstract, not its full methods or results; it does not provide details needed to assess task selection, evaluation conditions, or the uncertainty around reported rates.","The abstract reports results on InfraDev tasks and does not establish how well those results predict performance on other AI infrastructure work."],"mediaId":"ef598a94a45241da615d47c747793645385ce435ce6b4dc59dd3662be3275a37","correctionReason":null,"id":"bcb4875389a27651f30707b545fd7f466e60386ede3e3ae8ca8e8ca82b0d8527","publicationId":"4611878e-59e0-44c0-8bbf-c7969429b5cf","version":1,"sourceUrl":"https://arxiv.org/abs/2610.36589","sourceTitle":"InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-27","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-28T06:06:35.543Z","createdAt":"2026-11-13T22:18:24.416Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"ef598a94a45241da615d47c747793645385ce435ce6b4dc59dd3662be3275a37","kind":"article","url":"https://arxiv.org/abs/2610.36589","sourceUrl":"https://arxiv.org/abs/2610.36589","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-30T20:32:38.222Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.36589","checkedAt":"2026-11-14T00:26:07.746Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"ef598a94a45241da615d47c747793645385ce435ce6b4dc59dd3662be3275a37","confidence":0.9099999999999999,"provider":"typesafe","model":"jev-1.13.0","inputHash":"dc9e7f8cb5d13a2d9179b911f548f606de436be682a62d9a84ddc5d6e9073ab7","decidedAt":"2026-10-30T20:32:38.606Z"},"provenance":{"textHash":"99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc","observationId":"16fb9174289b446e0fc4743d59169958a0b792a8ee44875acebb371deadfc44a","retrievedAt":"2026-10-30T20:32:37.901Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"31916e47-7d2b-4367-83e7-321b0ebd7636"},"reviewer":{"model":"gpt-6-luna","runId":"3c2808c2-b20e-4289-816f-9f8a1c7ee103"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-13T22:18:24.416Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Deng, Yangshen","url":null,"kind":"unknown"},{"name":"Ruan, Jiaqi","url":null,"kind":"unknown"},{"name":"Huang, Yeqi","url":null,"kind":"unknown"},{"name":"Lau, Nick","url":null,"kind":"unknown"},{"name":"Sheng, Guangming","url":null,"kind":"unknown"},{"name":"Mai, Luo","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Deng, Yangshen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Deng, Yangshen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36589","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:07.746Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9","sourceUrl":"https://arxiv.org/abs/2610.36589","contentSha256":"c383c1b25a2076c86e9629015f6ee34adbdc4a2a63a40fd788c0661c8c3e9202","textHash":"99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc","observedAt":"2026-11-14T00:26:07.746Z"}},{"actor":{"name":"Ruan, Jiaqi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ruan, Jiaqi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36589","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:07.746Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9","sourceUrl":"https://arxiv.org/abs/2610.36589","contentSha256":"c383c1b25a2076c86e9629015f6ee34adbdc4a2a63a40fd788c0661c8c3e9202","textHash":"99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc","observedAt":"2026-11-14T00:26:07.746Z"}},{"actor":{"name":"Huang, Yeqi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Huang, Yeqi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36589","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:07.746Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9","sourceUrl":"https://arxiv.org/abs/2610.36589","contentSha256":"c383c1b25a2076c86e9629015f6ee34adbdc4a2a63a40fd788c0661c8c3e9202","textHash":"99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc","observedAt":"2026-11-14T00:26:07.746Z"}},{"actor":{"name":"Lau, Nick","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Lau, Nick","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36589","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:07.746Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9","sourceUrl":"https://arxiv.org/abs/2610.36589","contentSha256":"c383c1b25a2076c86e9629015f6ee34adbdc4a2a63a40fd788c0661c8c3e9202","textHash":"99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc","observedAt":"2026-11-14T00:26:07.746Z"}},{"actor":{"name":"Sheng, Guangming","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sheng, Guangming","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36589","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:07.746Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9","sourceUrl":"https://arxiv.org/abs/2610.36589","contentSha256":"c383c1b25a2076c86e9629015f6ee34adbdc4a2a63a40fd788c0661c8c3e9202","textHash":"99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc","observedAt":"2026-11-14T00:26:07.746Z"}},{"actor":{"name":"Mai, Luo","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mai, Luo","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36589","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:07.746Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9","sourceUrl":"https://arxiv.org/abs/2610.36589","contentSha256":"c383c1b25a2076c86e9629015f6ee34adbdc4a2a63a40fd788c0661c8c3e9202","textHash":"99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc","observedAt":"2026-11-14T00:26:07.746Z"}}],"participationCoverage":{"total":6,"returned":6,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.36589","snapshotId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9","contentSha256":"c383c1b25a2076c86e9629015f6ee34adbdc4a2a63a40fd788c0661c8c3e9202","textHash":"99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc","observedAt":"2026-11-14T00:26:07.746Z","basis":"publisher-page-metadata","sourceLinks":45,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/bcb4875389a27651f30707b545fd7f466e60386ede3e3ae8ca8e8ca82b0d8527/source"}},"canonicalUrl":"https://escapemeridian.com/readings/infradev-can-coding-agents-develop-ai-infrastructure-and-how-bcb48753","formats":{"html":"/readings/infradev-can-coding-agents-develop-ai-infrastructure-and-how-bcb48753","markdown":"/readings/infradev-can-coding-agents-develop-ai-infrastructure-and-how-bcb48753.md","json":"/readings/infradev-can-coding-agents-develop-ai-infrastructure-and-how-bcb48753.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-gpu","name":"Graphics processing unit","explanation":"A chip that performs many calculations at once; the main hardware used to train and run AI models.","referenceUrl":"https://en.wikipedia.org/wiki/Graphics_processing_unit"},{"id":"term-frontier-model","name":"Frontier model","explanation":"One of the most capable AI models available at a given time; the set shifts as models improve.","referenceUrl":"https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"},{"id":"time","name":"TIME","kind":"organization","context":"TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.","referenceUrls":["https://time.com/"],"path":"/entities/time","json":"/entities/time.json","markdown":"/entities/time.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"bcb4875389a27651f30707b545fd7f466e60386ede3e3ae8ca8e8ca82b0d8527","revision":"4611878e-59e0-44c0-8bbf-c7969429b5cf","revisionKind":"publication-id","title":"InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?","url":"https://escapemeridian.com/readings/infradev-can-coding-agents-develop-ai-infrastructure-and-how-bcb48753"},"source":{"type":"source","id":"https://arxiv.org/abs/2610.36589","revision":"99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc","revisionKind":"sha256-source-text","title":"InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?","url":"https://arxiv.org/abs/2610.36589"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"bcb4875389a27651f30707b545fd7f466e60386ede3e3ae8ca8e8ca82b0d8527","title":"InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?","argument":"Yangshen Deng and coauthors introduce InfraDev, a benchmark built from real-world changes to Megatron-LM, vLLM, and verl. Its 36 tasks ask coding agents to reproduce changes spanning many files and run them across many GPUs, rather than solve only single-kernel or single-GPU tasks. The authors say hidden probes check requirements that original tests miss, while prompts describe the interfaces without revealing the solution. Across six frontier models, the best resolves 38.9% of tasks; hidden probes reject 32% of submissions that pass every official test. With original pull request descriptions as prompts, no model resolves a task. The results suggest that passing a project’s tests can overstate whether an AI infrastructure change works, and that agents’ own tests often miss memory, time, or traffic. The authors also report that turning their failure analysis into an agent skill raises one frontier model’s resolved rate on Megatron-LM tasks by 10.5%. How well would these benchmark results predict coding agents’ performance on AI infrastructure changes outside the 36 tasks?","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence here is the paper’s abstract, not its full methods or results; it does not provide details needed to assess task selection, evaluation conditions, or the uncertainty around reported rates. The abstract reports results on InfraDev tasks and does not establish how well those results predict performance on other AI infrastructure work.","speaker":{"name":"Deng, Yangshen, Ruan, Jiaqi, Huang, Yeqi, Lau, Nick, Sheng, Guangming, Mai, Luo","roleAtSource":"Article author"},"source":{"title":"InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.36589","publishedAt":"2026-10-27","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce InfraDev, a benchmark of 36 tasks based on real-world pull requests to Megatron-LM, vLLM, and verl, covering training, inference, and reinforcement learning infrastructure development.","excerpt":"We introduce InfraDev, a benchmark of 36 tasks built from real-world pull requests to Megatron-LM, vLLM, and verl, covering infrastructure development for training, inference, and reinforcement learning.","url":"https://arxiv.org/abs/2610.36589","relationship":"Source-stated"},{"text":"InfraDev tasks ask agents to reimplement changes spanning many files and run across many GPUs.","excerpt":"Each task asks an agent to reimplement a real change that spans many files and must run correctly across many GPUs.","url":"https://arxiv.org/abs/2610.36589","relationship":"Source-stated"},{"text":"The authors report that hidden probes reject some submissions that pass all official tests, and that no model resolves a task when given original pull request descriptions as prompts.","excerpt":"the probes reject 32% of the submissions that pass every official test, and with the original pull request descriptions as prompts, no model resolves a single task.","url":"https://arxiv.org/abs/2610.36589","relationship":"Source-stated"},{"text":"Across six frontier models, the best resolves 38.9% of InfraDev tasks.","excerpt":"Across six frontier models, the best resolves only 38.9% of the tasks","url":"https://arxiv.org/abs/2610.36589","relationship":"Source-stated"},{"text":"The authors report that turning their findings into an agent skill raises a frontier model’s resolved rate on Megatron-LM tasks by 10.5%.","excerpt":"Turning these findings into an agent skill raises the resolved rate of a frontier model on the Megatron-LM tasks by 10.5%.","url":"https://arxiv.org/abs/2610.36589","relationship":"Source-stated"},{"text":"The findings suggest that passing official tests may not establish that a change meets requirements those tests leave unchecked.","excerpt":"the probes reject 32% of the submissions that pass every official test","url":"https://arxiv.org/abs/2610.36589","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.36589","checkedAt":"2026-11-14T00:26:07.746Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well would InfraDev’s results predict coding agents’ performance on AI infrastructure changes outside its 36 tasks?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?","summary":"Yangshen Deng and coauthors introduce InfraDev, a benchmark built from real-world changes to Megatron-LM, vLLM, and verl. Its 36 tasks ask coding agents to reproduce changes spanning many files and run them across many GPUs, rather than solve only single-kernel or single-GPU tasks. The authors say hidden probes check requirements that original tests miss, while prompts describe the interfaces without revealing the solution. Across six frontier models, the best resolves 38.9% of tasks; hidden probes reject 32% of submissions that pass every official test. With original pull request descriptions as prompts, no model resolves a task. The results suggest that passing a project’s tests can overstate whether an AI infrastructure change works, and that agents’ own tests often miss memory, time, or traffic. The authors also report that turning their failure analysis into an agent skill raises one frontier model’s resolved rate on Megatron-LM tasks by 10.5%. How well would these benchmark results predict coding agents’ performance on AI infrastructure changes outside the 36 tasks?","question":"How well would InfraDev’s results predict coding agents’ performance on AI infrastructure changes outside its 36 tasks?","claims":[{"text":"The authors introduce InfraDev, a benchmark of 36 tasks based on real-world pull requests to Megatron-LM, vLLM, and verl, covering training, inference, and reinforcement learning infrastructure development.","excerpt":"We introduce InfraDev, a benchmark of 36 tasks built from real-world pull requests to Megatron-LM, vLLM, and verl, covering infrastructure development for training, inference, and reinforcement learning.","support":"source-stated"},{"text":"InfraDev tasks ask agents to reimplement changes spanning many files and run across many GPUs.","excerpt":"Each task asks an agent to reimplement a real change that spans many files and must run correctly across many GPUs.","support":"source-stated"},{"text":"The authors report that hidden probes reject some submissions that pass all official tests, and that no model resolves a task when given original pull request descriptions as prompts.","excerpt":"the probes reject 32% of the submissions that pass every official test, and with the original pull request descriptions as prompts, no model resolves a single task.","support":"source-stated"},{"text":"Across six frontier models, the best resolves 38.9% of InfraDev tasks.","excerpt":"Across six frontier models, the best resolves only 38.9% of the tasks","support":"source-stated"},{"text":"The authors report that turning their findings into an agent skill raises a frontier model’s resolved rate on Megatron-LM tasks by 10.5%.","excerpt":"Turning these findings into an agent skill raises the resolved rate of a frontier model on the Megatron-LM tasks by 10.5%.","support":"source-stated"},{"text":"The findings suggest that passing official tests may not establish that a change meets requirements those tests leave unchecked.","excerpt":"the probes reject 32% of the submissions that pass every official test","support":"interpretation"}],"limitations":["The evidence here is the paper’s abstract, not its full methods or results; it does not provide details needed to assess task selection, evaluation conditions, or the uncertainty around reported rates.","The abstract reports results on InfraDev tasks and does not establish how well those results predict performance on other AI infrastructure work."],"mediaId":"ef598a94a45241da615d47c747793645385ce435ce6b4dc59dd3662be3275a37","correctionReason":null,"id":"bcb4875389a27651f30707b545fd7f466e60386ede3e3ae8ca8e8ca82b0d8527","publicationId":"4611878e-59e0-44c0-8bbf-c7969429b5cf","version":1,"sourceUrl":"https://arxiv.org/abs/2610.36589","sourceTitle":"InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-27","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-28T06:06:35.543Z","createdAt":"2026-11-13T22:18:24.416Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"ef598a94a45241da615d47c747793645385ce435ce6b4dc59dd3662be3275a37","kind":"article","url":"https://arxiv.org/abs/2610.36589","sourceUrl":"https://arxiv.org/abs/2610.36589","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-30T20:32:38.222Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.36589","checkedAt":"2026-11-14T00:26:07.746Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"ef598a94a45241da615d47c747793645385ce435ce6b4dc59dd3662be3275a37","confidence":0.9099999999999999,"provider":"typesafe","model":"jev-1.13.0","inputHash":"dc9e7f8cb5d13a2d9179b911f548f606de436be682a62d9a84ddc5d6e9073ab7","decidedAt":"2026-10-30T20:32:38.606Z"},"provenance":{"textHash":"99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc","observationId":"16fb9174289b446e0fc4743d59169958a0b792a8ee44875acebb371deadfc44a","retrievedAt":"2026-10-30T20:32:37.901Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"31916e47-7d2b-4367-83e7-321b0ebd7636"},"reviewer":{"model":"gpt-6-luna","runId":"3c2808c2-b20e-4289-816f-9f8a1c7ee103"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-13T22:18:24.416Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Deng, Yangshen","url":null,"kind":"unknown"},{"name":"Ruan, Jiaqi","url":null,"kind":"unknown"},{"name":"Huang, Yeqi","url":null,"kind":"unknown"},{"name":"Lau, Nick","url":null,"kind":"unknown"},{"name":"Sheng, Guangming","url":null,"kind":"unknown"},{"name":"Mai, Luo","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Deng, Yangshen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Deng, Yangshen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36589","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:07.746Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9","sourceUrl":"https://arxiv.org/abs/2610.36589","contentSha256":"c383c1b25a2076c86e9629015f6ee34adbdc4a2a63a40fd788c0661c8c3e9202","textHash":"99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc","observedAt":"2026-11-14T00:26:07.746Z"}},{"actor":{"name":"Ruan, Jiaqi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ruan, Jiaqi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36589","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:07.746Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9","sourceUrl":"https://arxiv.org/abs/2610.36589","contentSha256":"c383c1b25a2076c86e9629015f6ee34adbdc4a2a63a40fd788c0661c8c3e9202","textHash":"99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc","observedAt":"2026-11-14T00:26:07.746Z"}},{"actor":{"name":"Huang, Yeqi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Huang, Yeqi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36589","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:07.746Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9","sourceUrl":"https://arxiv.org/abs/2610.36589","contentSha256":"c383c1b25a2076c86e9629015f6ee34adbdc4a2a63a40fd788c0661c8c3e9202","textHash":"99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc","observedAt":"2026-11-14T00:26:07.746Z"}},{"actor":{"name":"Lau, Nick","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Lau, Nick","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36589","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:07.746Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9","sourceUrl":"https://arxiv.org/abs/2610.36589","contentSha256":"c383c1b25a2076c86e9629015f6ee34adbdc4a2a63a40fd788c0661c8c3e9202","textHash":"99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc","observedAt":"2026-11-14T00:26:07.746Z"}},{"actor":{"name":"Sheng, Guangming","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sheng, Guangming","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36589","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:07.746Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9","sourceUrl":"https://arxiv.org/abs/2610.36589","contentSha256":"c383c1b25a2076c86e9629015f6ee34adbdc4a2a63a40fd788c0661c8c3e9202","textHash":"99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc","observedAt":"2026-11-14T00:26:07.746Z"}},{"actor":{"name":"Mai, Luo","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mai, Luo","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.36589","locator":"head/meta[name=author|citation_author]"},"assertionId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:07.746Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9","sourceUrl":"https://arxiv.org/abs/2610.36589","contentSha256":"c383c1b25a2076c86e9629015f6ee34adbdc4a2a63a40fd788c0661c8c3e9202","textHash":"99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc","observedAt":"2026-11-14T00:26:07.746Z"}}],"participationCoverage":{"total":6,"returned":6,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.36589","snapshotId":"7e21081d3704c1a120ec5cfdaaad34b72ea8d3fb45d0623681aa1bd7c5c75ad9","contentSha256":"c383c1b25a2076c86e9629015f6ee34adbdc4a2a63a40fd788c0661c8c3e9202","textHash":"99ad5d386a738f3f8d506e948b64e62bfa5d83bbd342964323d5fd740eb22ddc","observedAt":"2026-11-14T00:26:07.746Z","basis":"publisher-page-metadata","sourceLinks":45,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/bcb4875389a27651f30707b545fd7f466e60386ede3e3ae8ca8e8ca82b0d8527/source"}}}],"audits":[{"publicationId":"4611878e-59e0-44c0-8bbf-c7969429b5cf","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading correctly credits InfraDev to Yangshen Deng and coauthors and attributes the benchmark design, results and agent-skill improvement to the authors.","excerpt":"Authors:Yangshen Deng, Jiaqi Ruan, Yeqi Huang, Nick Lau, Guangming Sheng, Luo Mai"},{"dimension":"summary","status":"uncertain","explanation":"The summary is faithful, but a reader without specialist background may not understand it. It never says that Megatron-LM, vLLM and verl are software projects for training, serving and reinforcement-learning large models. It also drops the source's training, inference and RL scope. Terms such as single-kernel, GPUs, hidden probes and agent skill appear without plain explanation or term-library tooltips. The smallest repair is to say what the three projects do and add tooltips for these terms. Smaller issues: the summary repeats the question verbatim, and dropping the source's 'only' before 38.9% slightly softens the authors' framing.","excerpt":"covering infrastructure development for training, inference, and reinforcement learning"},{"dimension":"date","status":"pass","explanation":"The originalPublished date of 2026-10-27 matches the v1 arXiv submission date. The 2026-10-29 feed date is later and is correctly not used.","excerpt":"[Submitted on 27 Oct 2026]"},{"dimension":"support","status":"pass","explanation":"Every claim matches the abstract: the 36 tasks, multi-file multi-GPU changes, the 38.9% best resolve rate, the 32% of test-passing submissions rejected by probes, zero resolves with the original PR descriptions, and the 10.5% gain from the agent skill. The interpretation that passing tests can overstate correctness is hedged and supported by the abstract's own framing. 'Often miss' fairly paraphrases 'rarely check'. The limitations and the question do not overreach.","excerpt":"grade with tests and prompts that misjudge both correct and incorrect patches"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the arXiv abstract page for this paper. It is the most direct way for a reader to reach the source.","excerpt":null}],"createdAt":"2026-11-13T22:18:24.416Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}