· Feb 14, 2025
METR reports agents speed up GPU kernels, with limits to the tests
Markdown · JSON · Original source
’s post, published February 14, 2025, argues that can improve kernel performance in its modified KernelBench tests, while stressing that results depend on how the evaluation is designed. Using the same models as the original KernelBench paper, METR reports higher average speedup with KernelAgent and attributes the difference to scaffolding and greater . Its best result across models on each problem was higher still. METR says these results do not establish that language models can automate kernel engineering done by human experts. The benchmark uses single-GPU tasks and does not represent frontier applications involving communication between GPUs; METR also reports that its agents could not adapt FlashAttention 2 to new constraints. The results therefore leave open how well measured speedups transfer to real-world GPU kernel engineering. METR concludes that the work may be relevant to assessing AI capabilities, while judging its likely effect on algorithmic research outside to be small.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
· Article
Source details
Original source date: Feb 14, 2025 (2025-02-14T08:00:00.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-15T09:53:23.662Z. First observed: 2026-10-18T03:39:21.984Z. Retrieved: 2026-10-23T16:43:45.003Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Relevance
Escape Meridian’s relevance screen could not settle whether this source bears on the questions it tracks. Such sources are read rather than held, and this reading was checked against its source like any other. The uncertainty is EM’s, not a claim of the source.
Limits
Text as retrieved on October 23, 2026; originally published February 14, 2025. The results are from modified KernelBench tasks; notes that they do not perfectly match real-world kernel engineering. METR reports no comparison against humans given the same tasks. The results therefore do not establish how agents compare with human kernel engineers.
Open question
How well would these speedups carry over to real-world kernel engineering tasks, including work involving multiple GPUs or training?
Source-stated
METR says KernelAgent achieved higher average speedup than the original KernelBench results using the same models, and attributes the difference to scaffolding and greater test-time compute.
Using the same models as the original KernelBench paper, KernelAgent achieves a speedup of 1.81x vs KernelBench's 1.05x.
Source-stated
METR reports that the best result across models on each problem achieved an average speedup of 2.01x.
taking the best of all models per problem using KernelAgent achieved 2.01x speedup
Source-stated
METR says the benchmark covers single-GPU inference workloads and does not represent all real-world GPU kernel engineering.
KernelBench problems all run on a single GPU, which makes it accessible to researchers but not representative of frontier applications, which also include communication between GPUs.
Source-stated
METR says its results do not directly imply that language models can automate kernel engineering performed by human experts.
our results do not directly imply that language models can automate the kernel engineering work done by current human experts.
Source-stated
METR reports that its agents could not adapt FlashAttention 2 to new constraints when prompted.
Empirically, our agents were unable to adapt FlashAttention 2 to new constraints when we prompted them to do so.
Source-stated
METR judges that the likely effect of this work on algorithmic research outside frontier labs is small.
We believe this effect is likely small (as the open source state-of-the-art substantially lags behind frontier labs)
Source and contributor context
METR · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Graphics processing unit
A chip that performs many calculations at once; the main hardware used to train and run AI models.
AI agent
An AI system that chooses and carries out actions toward a goal, often using software tools.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Test-time compute
Extra computation a model spends while answering, for example by reasoning longer, rather than during training.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
METR
METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Editorial provenance
Version 1. Publication ID: 3db13181-6349-0269-5090-116855fb860b.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-15T09:48:39.771Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_screen_uncertain.
Observation: f20ded95f99f2b6fb4b3f888218e8e067f578a886ce892e8230398b965680358. Source text hash: 2e85f55900974ac231551054586db0c9f4e23f9a7d27b39595afb61671d3389b.
Revision and source-check history
Version 1 · 2026-11-15T09:53:23.662Z
METR reports agents speed up GPU kernels, with limits to the tests
’s post, published February 14, 2025, argues that can improve kernel performance in its modified KernelBench tests, while stressing that results depend on how the evaluation is designed. Using the same models as the original KernelBench paper, METR reports higher average speedup with KernelAgent and attributes the difference to scaffolding and greater . Its best result across models on each problem was higher still. METR says these results do not establish that language models can automate kernel engineering done by human experts. The benchmark uses single-GPU tasks and does not represent frontier applications involving communication between GPUs; METR also reports that its agents could not adapt FlashAttention 2 to new constraints. The results therefore leave open how well measured speedups transfer to real-world GPU kernel engineering. METR concludes that the work may be relevant to assessing AI capabilities, while judging its likely effect on algorithmic research outside to be small.
Original source
· Article
Source details
Original source date: Feb 14, 2025 (2025-02-14T08:00:00.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-15T09:53:23.662Z. First observed: 2026-10-18T03:39:21.984Z. Retrieved: 2026-10-23T16:43:45.003Z.
Relevance
Escape Meridian’s relevance screen could not settle whether this source bears on the questions it tracks. Such sources are read rather than held, and this reading was checked against its source like any other. The uncertainty is EM’s, not a claim of the source.
Limits
Text as retrieved on October 23, 2026; originally published February 14, 2025. The results are from modified KernelBench tasks; notes that they do not perfectly match real-world kernel engineering. METR reports no comparison against humans given the same tasks. The results therefore do not establish how agents compare with human kernel engineers.
Open question
How well would these speedups carry over to real-world kernel engineering tasks, including work involving multiple GPUs or training?
Source-stated
METR says KernelAgent achieved higher average speedup than the original KernelBench results using the same models, and attributes the difference to scaffolding and greater test-time compute.
Using the same models as the original KernelBench paper, KernelAgent achieves a speedup of 1.81x vs KernelBench's 1.05x.
Source-stated
METR reports that the best result across models on each problem achieved an average speedup of 2.01x.
taking the best of all models per problem using KernelAgent achieved 2.01x speedup
Source-stated
METR says the benchmark covers single-GPU inference workloads and does not represent all real-world GPU kernel engineering.
KernelBench problems all run on a single GPU, which makes it accessible to researchers but not representative of frontier applications, which also include communication between GPUs.
Source-stated
METR says its results do not directly imply that language models can automate kernel engineering performed by human experts.
our results do not directly imply that language models can automate the kernel engineering work done by current human experts.
Source-stated
METR reports that its agents could not adapt FlashAttention 2 to new constraints when prompted.
Empirically, our agents were unable to adapt FlashAttention 2 to new constraints when we prompted them to do so.
Source-stated
METR judges that the likely effect of this work on algorithmic research outside frontier labs is small.
We believe this effect is likely small (as the open source state-of-the-art substantially lags behind frontier labs)
Source and contributor context
METR · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Graphics processing unit
A chip that performs many calculations at once; the main hardware used to train and run AI models.
AI agent
An AI system that chooses and carries out actions toward a goal, often using software tools.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Test-time compute
Extra computation a model spends while answering, for example by reasoning longer, rather than during training.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
METR
METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Editorial provenance
Version 1. Publication ID: 3db13181-6349-0269-5090-116855fb860b.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-15T09:48:39.771Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_screen_uncertain.
Observation: f20ded95f99f2b6fb4b3f888218e8e067f578a886ce892e8230398b965680358. Source text hash: 2e85f55900974ac231551054586db0c9f4e23f9a7d27b39595afb61671d3389b.
uncertain · 2026-11-15T09:48:39.771Z · anthropic/claude-opus-5.5
Publication: 3db13181-6349-0269-5090-116855fb860b
attribution — pass: The reading consistently attributes the findings, caveats and judgments to METR, which published this single blog post. Claims are framed as what METR reports, says or judges, not as established fact.
In this new work, we focus specifically on measuring AIs’ ability to optimize compute kernels
summary — uncertain: The central term 'GPU kernel' or 'compute kernel' is never explained, and the definitions list has no entry for it. The reading also leaves 'scaffolding', KernelAgent (METR's own agent) and FlashAttention 2 unexplained. It omits METR's stated reason the work matters, AI automating AI R&D, as well as its finding that the speedup doubled over 6 months. A non-specialist cannot see why kernel speedups matter. Smallest repair: add a plain-language clause on what kernels are, note METR's AI R&D framing, and suggest a term-library entry for 'compute kernel'.
Compute kernels are specialized, low-level programs that serve as the building blocks for AI training and inference.
date — pass: evidence.originalPublished is 2025-02-14T08:00Z, so its UTC day is February 14, 2025. This matches the reading and the URL path. The source is a publisher page, not an archive copy. The reading uses no later events and no hindsight framing.
support — pass: Each claim matches the source text, including the 1.81x vs 1.05x result, the 2.01x best-of-models figure and the single-GPU limit. The source also states the inference-only scope, METR's statement that automation of expert work is not implied, the FlashAttention 2 failure and the 'likely small' effect. The question's mention of training is grounded in the inference-only limitation. The note that there was no human comparison is source-stated.
KernelBench only measures inference workloads, not backpropagation for training.
media — pass: No media was selected, and no verified candidates were supplied. Omitting media is appropriate.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
METR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRAn AI system that chooses and carries out actions toward a goal, often using software tools.
Definition referenceA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceExtra computation a model spends while answering, for example by reasoning longer, rather than during training.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition referenceA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition referenceMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRAn AI system that chooses and carries out actions toward a goal, often using software tools.
Definition referenceA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceExtra computation a model spends while answering, for example by reasoning longer, rather than during training.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceMETR studies what AI systems can do and how reliably they complete tasks, including work with human developers.
Explore METRA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition referenceA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition reference