{"id":"a05ce875cfa09a4ff00327b61a5f10d86196ad4015707909ac63d1747921d7a3","title":"METR reports agents speed up GPU kernels, with limits to the tests","argument":"METR’s post, published February 14, 2025, argues that AI agents can improve GPU kernel performance in its modified KernelBench tests, while stressing that benchmark results depend on how the evaluation is designed. Using the same models as the original KernelBench paper, METR reports higher average speedup with KernelAgent and attributes the difference to scaffolding and greater test-time compute. Its best result across models on each problem was higher still. METR says these results do not establish that language models can automate kernel engineering done by human experts. The benchmark uses single-GPU tasks and does not represent frontier applications involving communication between GPUs; METR also reports that its agents could not adapt FlashAttention 2 to new constraints. The results therefore leave open how well measured speedups transfer to real-world GPU kernel engineering. METR concludes that the work may be relevant to assessing AI capabilities, while judging its likely effect on algorithmic research outside frontier labs to be small.","topicIds":["work-and-business--jobs-and-occupations","job-displacement-and-creation"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 23, 2026; originally published February 14, 2025. The benchmark results are from modified KernelBench tasks; METR notes that they do not perfectly match real-world GPU kernel engineering. METR reports no comparison against humans given the same tasks. The results therefore do not establish how agents compare with human kernel engineers.","speaker":{"name":"METR","roleAtSource":"Publisher"},"source":{"title":"Measuring Automated Kernel Engineering","publisher":"METR","format":"Article","url":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","publishedAt":"2025-02-14T08:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"METR says KernelAgent achieved higher average speedup than the original KernelBench results using the same models, and attributes the difference to scaffolding and greater test-time compute.","excerpt":"Using the same models as the original KernelBench paper, KernelAgent achieves a speedup of 1.81x vs KernelBench's 1.05x.","url":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","relationship":"Source-stated"},{"text":"METR reports that the best result across models on each problem achieved an average speedup of 2.01x.","excerpt":"taking the best of all models per problem using KernelAgent achieved 2.01x speedup","url":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","relationship":"Source-stated"},{"text":"METR says the benchmark covers single-GPU inference workloads and does not represent all real-world GPU kernel engineering.","excerpt":"KernelBench problems all run on a single GPU, which makes it accessible to researchers but not representative of frontier applications, which also include communication between GPUs.","url":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","relationship":"Source-stated"},{"text":"METR says its results do not directly imply that language models can automate kernel engineering performed by human experts.","excerpt":"our results do not directly imply that language models can automate the kernel engineering work done by current human experts.","url":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","relationship":"Source-stated"},{"text":"METR reports that its agents could not adapt FlashAttention 2 to new constraints when prompted.","excerpt":"Empirically, our agents were unable to adapt FlashAttention 2 to new constraints when we prompted them to do so.","url":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","relationship":"Source-stated"},{"text":"METR judges that the likely effect of this work on algorithmic research outside frontier labs is small.","excerpt":"We believe this effect is likely small (as the open source state-of-the-art substantially lags behind frontier labs)","url":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well would these speedups carry over to real-world GPU kernel engineering tasks, including work involving multiple GPUs or training?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"METR reports agents speed up GPU kernels, with limits to the tests","summary":"METR’s post, published February 14, 2025, argues that AI agents can improve GPU kernel performance in its modified KernelBench tests, while stressing that benchmark results depend on how the evaluation is designed. Using the same models as the original KernelBench paper, METR reports higher average speedup with KernelAgent and attributes the difference to scaffolding and greater test-time compute. Its best result across models on each problem was higher still. METR says these results do not establish that language models can automate kernel engineering done by human experts. The benchmark uses single-GPU tasks and does not represent frontier applications involving communication between GPUs; METR also reports that its agents could not adapt FlashAttention 2 to new constraints. The results therefore leave open how well measured speedups transfer to real-world GPU kernel engineering. METR concludes that the work may be relevant to assessing AI capabilities, while judging its likely effect on algorithmic research outside frontier labs to be small.","question":"How well would these speedups carry over to real-world GPU kernel engineering tasks, including work involving multiple GPUs or training?","claims":[{"text":"METR says KernelAgent achieved higher average speedup than the original KernelBench results using the same models, and attributes the difference to scaffolding and greater test-time compute.","excerpt":"Using the same models as the original KernelBench paper, KernelAgent achieves a speedup of 1.81x vs KernelBench's 1.05x.","support":"source-stated"},{"text":"METR reports that the best result across models on each problem achieved an average speedup of 2.01x.","excerpt":"taking the best of all models per problem using KernelAgent achieved 2.01x speedup","support":"source-stated"},{"text":"METR says the benchmark covers single-GPU inference workloads and does not represent all real-world GPU kernel engineering.","excerpt":"KernelBench problems all run on a single GPU, which makes it accessible to researchers but not representative of frontier applications, which also include communication between GPUs.","support":"source-stated"},{"text":"METR says its results do not directly imply that language models can automate kernel engineering performed by human experts.","excerpt":"our results do not directly imply that language models can automate the kernel engineering work done by current human experts.","support":"source-stated"},{"text":"METR reports that its agents could not adapt FlashAttention 2 to new constraints when prompted.","excerpt":"Empirically, our agents were unable to adapt FlashAttention 2 to new constraints when we prompted them to do so.","support":"source-stated"},{"text":"METR judges that the likely effect of this work on algorithmic research outside frontier labs is small.","excerpt":"We believe this effect is likely small (as the open source state-of-the-art substantially lags behind frontier labs)","support":"source-stated"}],"limitations":["Text as retrieved on October 23, 2026; originally published February 14, 2025.","The benchmark results are from modified KernelBench tasks; METR notes that they do not perfectly match real-world GPU kernel engineering.","METR reports no comparison against humans given the same tasks. The results therefore do not establish how agents compare with human kernel engineers."],"mediaId":null,"correctionReason":null,"id":"a05ce875cfa09a4ff00327b61a5f10d86196ad4015707909ac63d1747921d7a3","publicationId":"3db13181-6349-0269-5090-116855fb860b","version":1,"sourceUrl":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","sourceTitle":"Measuring Automated Kernel Engineering","publisher":"METR","publishedAt":"2025-02-14T08:00:00.000Z","originalPublished":{"value":"2025-02-14T08:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T03:39:21.984Z","createdAt":"2026-11-15T09:53:23.662Z","historical":true,"entityIds":["metr"],"topicIds":["work-and-business--jobs-and-occupations","job-displacement-and-creation"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"ac51f875800f2f3330964f72a098002eeb7d278a3ecd7fe307a32a7993ae2e4b","reasonCodes":["bridge_screen_uncertain"],"rubricVersion":"bridge-screen-v5","confidence":0.43,"provider":"typesafe","model":"jev-1.13.0","decidedAt":"2026-10-29T14:29:47.234Z"},"provenance":{"textHash":"2e85f55900974ac231551054586db0c9f4e23f9a7d27b39595afb61671d3389b","observationId":"f20ded95f99f2b6fb4b3f888218e8e067f578a886ce892e8230398b965680358","retrievedAt":"2026-10-23T16:43:45.003Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"bd87b617-9bb8-c498-9188-4d5aacc841ef"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"822da55f-d34b-c3af-a4a6-9a249b5d7e24"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-15T09:48:39.771Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/metr-reports-agents-speed-up-gpu-kernels-with-limits-to-the-a05ce875","formats":{"html":"/readings/metr-reports-agents-speed-up-gpu-kernels-with-limits-to-the-a05ce875","markdown":"/readings/metr-reports-agents-speed-up-gpu-kernels-with-limits-to-the-a05ce875.md","json":"/readings/metr-reports-agents-speed-up-gpu-kernels-with-limits-to-the-a05ce875.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"relevanceNote":"Escape Meridian’s relevance screen could not settle whether this source bears on the questions it tracks. Such sources are read rather than held, and this reading was checked against its source like any other. The uncertainty is EM’s, not a claim of the source.","readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-gpu","name":"Graphics processing unit","explanation":"A chip that performs many calculations at once; the main hardware used to train and run AI models.","referenceUrl":"https://en.wikipedia.org/wiki/Graphics_processing_unit"},{"id":"term-ai-agent","name":"AI agent","explanation":"An AI system that chooses and carries out actions toward a goal, often using software tools.","referenceUrl":"https://www.ibm.com/think/topics/ai-agents"},{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-test-time-compute","name":"Test-time compute","explanation":"Extra computation a model spends while answering, for example by reasoning longer, rather than during training.","referenceUrl":"https://arxiv.org/abs/2408.03314"},{"id":"term-frontier-model","name":"Frontier model","explanation":"One of the most capable AI models available at a given time; the set shifts as models improve.","referenceUrl":"https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper"}],"entities":[{"id":"metr","name":"METR","kind":"organization","context":"METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.","referenceUrls":["https://metr.org/"],"path":"/entities/metr","json":"/entities/metr.json","markdown":"/entities/metr.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"a05ce875cfa09a4ff00327b61a5f10d86196ad4015707909ac63d1747921d7a3","revision":"3db13181-6349-0269-5090-116855fb860b","revisionKind":"publication-id","title":"METR reports agents speed up GPU kernels, with limits to the tests","url":"https://escapemeridian.com/readings/metr-reports-agents-speed-up-gpu-kernels-with-limits-to-the-a05ce875"},"source":{"type":"source","id":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","revision":"2e85f55900974ac231551054586db0c9f4e23f9a7d27b39595afb61671d3389b","revisionKind":"sha256-source-text","title":"Measuring Automated Kernel Engineering","url":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"a05ce875cfa09a4ff00327b61a5f10d86196ad4015707909ac63d1747921d7a3","title":"METR reports agents speed up GPU kernels, with limits to the tests","argument":"METR’s post, published February 14, 2025, argues that AI agents can improve GPU kernel performance in its modified KernelBench tests, while stressing that benchmark results depend on how the evaluation is designed. Using the same models as the original KernelBench paper, METR reports higher average speedup with KernelAgent and attributes the difference to scaffolding and greater test-time compute. Its best result across models on each problem was higher still. METR says these results do not establish that language models can automate kernel engineering done by human experts. The benchmark uses single-GPU tasks and does not represent frontier applications involving communication between GPUs; METR also reports that its agents could not adapt FlashAttention 2 to new constraints. The results therefore leave open how well measured speedups transfer to real-world GPU kernel engineering. METR concludes that the work may be relevant to assessing AI capabilities, while judging its likely effect on algorithmic research outside frontier labs to be small.","topicIds":["work-and-business--jobs-and-occupations","job-displacement-and-creation"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 23, 2026; originally published February 14, 2025. The benchmark results are from modified KernelBench tasks; METR notes that they do not perfectly match real-world GPU kernel engineering. METR reports no comparison against humans given the same tasks. The results therefore do not establish how agents compare with human kernel engineers.","speaker":{"name":"METR","roleAtSource":"Publisher"},"source":{"title":"Measuring Automated Kernel Engineering","publisher":"METR","format":"Article","url":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","publishedAt":"2025-02-14T08:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"METR says KernelAgent achieved higher average speedup than the original KernelBench results using the same models, and attributes the difference to scaffolding and greater test-time compute.","excerpt":"Using the same models as the original KernelBench paper, KernelAgent achieves a speedup of 1.81x vs KernelBench's 1.05x.","url":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","relationship":"Source-stated"},{"text":"METR reports that the best result across models on each problem achieved an average speedup of 2.01x.","excerpt":"taking the best of all models per problem using KernelAgent achieved 2.01x speedup","url":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","relationship":"Source-stated"},{"text":"METR says the benchmark covers single-GPU inference workloads and does not represent all real-world GPU kernel engineering.","excerpt":"KernelBench problems all run on a single GPU, which makes it accessible to researchers but not representative of frontier applications, which also include communication between GPUs.","url":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","relationship":"Source-stated"},{"text":"METR says its results do not directly imply that language models can automate kernel engineering performed by human experts.","excerpt":"our results do not directly imply that language models can automate the kernel engineering work done by current human experts.","url":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","relationship":"Source-stated"},{"text":"METR reports that its agents could not adapt FlashAttention 2 to new constraints when prompted.","excerpt":"Empirically, our agents were unable to adapt FlashAttention 2 to new constraints when we prompted them to do so.","url":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","relationship":"Source-stated"},{"text":"METR judges that the likely effect of this work on algorithmic research outside frontier labs is small.","excerpt":"We believe this effect is likely small (as the open source state-of-the-art substantially lags behind frontier labs)","url":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well would these speedups carry over to real-world GPU kernel engineering tasks, including work involving multiple GPUs or training?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"METR reports agents speed up GPU kernels, with limits to the tests","summary":"METR’s post, published February 14, 2025, argues that AI agents can improve GPU kernel performance in its modified KernelBench tests, while stressing that benchmark results depend on how the evaluation is designed. Using the same models as the original KernelBench paper, METR reports higher average speedup with KernelAgent and attributes the difference to scaffolding and greater test-time compute. Its best result across models on each problem was higher still. METR says these results do not establish that language models can automate kernel engineering done by human experts. The benchmark uses single-GPU tasks and does not represent frontier applications involving communication between GPUs; METR also reports that its agents could not adapt FlashAttention 2 to new constraints. The results therefore leave open how well measured speedups transfer to real-world GPU kernel engineering. METR concludes that the work may be relevant to assessing AI capabilities, while judging its likely effect on algorithmic research outside frontier labs to be small.","question":"How well would these speedups carry over to real-world GPU kernel engineering tasks, including work involving multiple GPUs or training?","claims":[{"text":"METR says KernelAgent achieved higher average speedup than the original KernelBench results using the same models, and attributes the difference to scaffolding and greater test-time compute.","excerpt":"Using the same models as the original KernelBench paper, KernelAgent achieves a speedup of 1.81x vs KernelBench's 1.05x.","support":"source-stated"},{"text":"METR reports that the best result across models on each problem achieved an average speedup of 2.01x.","excerpt":"taking the best of all models per problem using KernelAgent achieved 2.01x speedup","support":"source-stated"},{"text":"METR says the benchmark covers single-GPU inference workloads and does not represent all real-world GPU kernel engineering.","excerpt":"KernelBench problems all run on a single GPU, which makes it accessible to researchers but not representative of frontier applications, which also include communication between GPUs.","support":"source-stated"},{"text":"METR says its results do not directly imply that language models can automate kernel engineering performed by human experts.","excerpt":"our results do not directly imply that language models can automate the kernel engineering work done by current human experts.","support":"source-stated"},{"text":"METR reports that its agents could not adapt FlashAttention 2 to new constraints when prompted.","excerpt":"Empirically, our agents were unable to adapt FlashAttention 2 to new constraints when we prompted them to do so.","support":"source-stated"},{"text":"METR judges that the likely effect of this work on algorithmic research outside frontier labs is small.","excerpt":"We believe this effect is likely small (as the open source state-of-the-art substantially lags behind frontier labs)","support":"source-stated"}],"limitations":["Text as retrieved on October 23, 2026; originally published February 14, 2025.","The benchmark results are from modified KernelBench tasks; METR notes that they do not perfectly match real-world GPU kernel engineering.","METR reports no comparison against humans given the same tasks. The results therefore do not establish how agents compare with human kernel engineers."],"mediaId":null,"correctionReason":null,"id":"a05ce875cfa09a4ff00327b61a5f10d86196ad4015707909ac63d1747921d7a3","publicationId":"3db13181-6349-0269-5090-116855fb860b","version":1,"sourceUrl":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","sourceTitle":"Measuring Automated Kernel Engineering","publisher":"METR","publishedAt":"2025-02-14T08:00:00.000Z","originalPublished":{"value":"2025-02-14T08:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T03:39:21.984Z","createdAt":"2026-11-15T09:53:23.662Z","historical":true,"entityIds":["metr"],"topicIds":["work-and-business--jobs-and-occupations","job-displacement-and-creation"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"ac51f875800f2f3330964f72a098002eeb7d278a3ecd7fe307a32a7993ae2e4b","reasonCodes":["bridge_screen_uncertain"],"rubricVersion":"bridge-screen-v5","confidence":0.43,"provider":"typesafe","model":"jev-1.13.0","decidedAt":"2026-10-29T14:29:47.234Z"},"provenance":{"textHash":"2e85f55900974ac231551054586db0c9f4e23f9a7d27b39595afb61671d3389b","observationId":"f20ded95f99f2b6fb4b3f888218e8e067f578a886ce892e8230398b965680358","retrievedAt":"2026-10-23T16:43:45.003Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"bd87b617-9bb8-c498-9188-4d5aacc841ef"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"822da55f-d34b-c3af-a4a6-9a249b5d7e24"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-15T09:48:39.771Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"3db13181-6349-0269-5090-116855fb860b","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading consistently attributes the findings, caveats and judgments to METR, which published this single blog post. Claims are framed as what METR reports, says or judges, not as established fact.","excerpt":"In this new work, we focus specifically on measuring AIs’ ability to optimize compute kernels"},{"dimension":"summary","status":"uncertain","explanation":"The central term 'GPU kernel' or 'compute kernel' is never explained, and the definitions list has no entry for it. The reading also leaves 'scaffolding', KernelAgent (METR's own agent) and FlashAttention 2 unexplained. It omits METR's stated reason the work matters, AI automating AI R&D, as well as its finding that the speedup doubled over 6 months. A non-specialist cannot see why kernel speedups matter. Smallest repair: add a plain-language clause on what kernels are, note METR's AI R&D framing, and suggest a term-library entry for 'compute kernel'.","excerpt":"Compute kernels are specialized, low-level programs that serve as the building blocks for AI training and inference."},{"dimension":"date","status":"pass","explanation":"evidence.originalPublished is 2025-02-14T08:00Z, so its UTC day is February 14, 2025. This matches the reading and the URL path. The source is a publisher page, not an archive copy. The reading uses no later events and no hindsight framing.","excerpt":null},{"dimension":"support","status":"pass","explanation":"Each claim matches the source text, including the 1.81x vs 1.05x result, the 2.01x best-of-models figure and the single-GPU limit. The source also states the inference-only scope, METR's statement that automation of expert work is not implied, the FlashAttention 2 failure and the 'likely small' effect. The question's mention of training is grounded in the inference-only limitation. The note that there was no human comparison is source-stated.","excerpt":"KernelBench only measures inference workloads, not backpropagation for training."},{"dimension":"media","status":"pass","explanation":"No media was selected, and no verified candidates were supplied. Omitting media is appropriate.","excerpt":null}],"createdAt":"2026-11-15T09:48:39.771Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}