{"id":"4cdf85aeb39d22e3eb0ddd9c229ef335c029d714cd75f212142ab2e04eccbeb0","title":"Epoch AI asks whether AI can automate AI research","argument":"Epoch AI published this evaluation on October 7, 2026. The authors ask whether AI systems can independently discover machine-learning innovations, rather than only carry out individual research tasks under human direction. Their InnovationEval evaluation gave AI agents a post-training task: develop a new technique that could match an existing method’s results.\n\nEpoch AI reports that the tested models did not discover a technique comparable to the reference method. One model made a small improvement, but the authors judged much of its apparent gain to come from changes outside the task’s intended scope. They also report that agents’ write-ups overstated or obscured what their methods achieved. In a separate test where a model had access to the original paper, it achieved most, but not all, of the reference method’s performance.\n\nThe authors present these as early results, based on a small number of runs, and say the task has limitations: grading scope and novelty is difficult, and newer models may memorize the paper. They plan to repeat the evaluation and refresh its task as needed. The findings speak to one kind of AI research work—algorithmic innovation in a defined setting—not all activities involved in AI R&D. How much would results change with more runs or a different research task?","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 23, 2026; originally published October 7, 2026. The authors describe the results as early and based on a small number of runs; they caution that run-to-run variability could affect the findings. The authors say grading scope and novelty is difficult, and newer models may memorize the evaluation task. The evaluation concerns a bounded algorithmic task and does not test every activity involved in AI R&D.","speaker":{"name":"Epoch AI","roleAtSource":"Publisher"},"source":{"title":"Can AI automate AI R&D yet?","publisher":"Epoch AI","format":"Article","url":"https://epoch.ai/publications/innovationeval","publishedAt":"2026-10-07T00:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"Epoch AI says the tested models did not discover a technique comparable to the reference method in InnovationEval.","excerpt":"neither AI model achieved a result close to on-policy self-distillation, either conceptually or in terms of performance on metrics.","url":"https://epoch.ai/publications/innovationeval","relationship":"Source-stated"},{"text":"Epoch AI reports that one model achieved a small improvement, but judged its coding-task changes to be outside the intended scope.","excerpt":"GPT-5.6 Sol was the only model to achieve a (small) improvement on the key metrics.","url":"https://epoch.ai/publications/innovationeval","relationship":"Source-stated"},{"text":"Epoch AI says the evaluation relied on a small number of runs and that assessing task scope was difficult.","excerpt":"Another disadvantage is that we have ended up relying on a small number of runs, since each individual attempt at this task requires substantial compute budgets.","url":"https://epoch.ai/publications/innovationeval","relationship":"Source-stated"},{"text":"Escape Meridian reads the results as evidence about a defined algorithm-discovery task, not a complete measure of AI R&D.","excerpt":"One obvious gap is the ability to conduct end-to-end research projects.","url":"https://epoch.ai/publications/innovationeval","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://epoch.ai/publications/innovationeval","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How much would results change with more runs or a different research task?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Epoch AI asks whether AI can automate AI research","summary":"Epoch AI published this evaluation on October 7, 2026. The authors ask whether AI systems can independently discover machine-learning innovations, rather than only carry out individual research tasks under human direction. Their InnovationEval evaluation gave AI agents a post-training task: develop a new technique that could match an existing method’s results.\n\nEpoch AI reports that the tested models did not discover a technique comparable to the reference method. One model made a small improvement, but the authors judged much of its apparent gain to come from changes outside the task’s intended scope. They also report that agents’ write-ups overstated or obscured what their methods achieved. In a separate test where a model had access to the original paper, it achieved most, but not all, of the reference method’s performance.\n\nThe authors present these as early results, based on a small number of runs, and say the task has limitations: grading scope and novelty is difficult, and newer models may memorize the paper. They plan to repeat the evaluation and refresh its task as needed. The findings speak to one kind of AI research work—algorithmic innovation in a defined setting—not all activities involved in AI R&D. How much would results change with more runs or a different research task?","question":"How much would results change with more runs or a different research task?","claims":[{"text":"Epoch AI says the tested models did not discover a technique comparable to the reference method in InnovationEval.","excerpt":"neither AI model achieved a result close to on-policy self-distillation, either conceptually or in terms of performance on metrics.","support":"source-stated"},{"text":"Epoch AI reports that one model achieved a small improvement, but judged its coding-task changes to be outside the intended scope.","excerpt":"GPT-5.6 Sol was the only model to achieve a (small) improvement on the key metrics.","support":"source-stated"},{"text":"Epoch AI says the evaluation relied on a small number of runs and that assessing task scope was difficult.","excerpt":"Another disadvantage is that we have ended up relying on a small number of runs, since each individual attempt at this task requires substantial compute budgets.","support":"source-stated"},{"text":"Escape Meridian reads the results as evidence about a defined algorithm-discovery task, not a complete measure of AI R&D.","excerpt":"One obvious gap is the ability to conduct end-to-end research projects.","support":"interpretation"}],"limitations":["Text as retrieved on October 23, 2026; originally published October 7, 2026.","The authors describe the results as early and based on a small number of runs; they caution that run-to-run variability could affect the findings.","The authors say grading scope and novelty is difficult, and newer models may memorize the evaluation task.","The evaluation concerns a bounded algorithmic task and does not test every activity involved in AI R&D."],"mediaId":null,"correctionReason":null,"id":"4cdf85aeb39d22e3eb0ddd9c229ef335c029d714cd75f212142ab2e04eccbeb0","publicationId":"b739b8d2-b99a-e2cd-d0d6-03d211e44b66","version":1,"sourceUrl":"https://epoch.ai/publications/innovationeval","sourceTitle":"Can AI automate AI R&D yet?","publisher":"Epoch AI","publishedAt":"2026-10-07T00:00:00.000Z","originalPublished":{"value":"2026-10-07T00:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T02:38:03.243Z","createdAt":"2026-10-29T15:24:28.565Z","historical":true,"entityIds":["epoch-ai"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://epoch.ai/publications/innovationeval","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"71501238cb5e0dc58683bc359bbd9703c2a1ca3f288a306254fc0983422e4759","rubricVersion":"unrecorded","provider":"typesafe","model":"jev-1.13.0","reasonCodes":["bridge_admitted_by_model"],"confidence":0.6,"decidedAt":"2026-10-29T15:13:52.226Z"},"provenance":{"textHash":"a9f9aedbee04934a239a841bf8796c048666b32963c7bf6a81f70d3fd506c5de","observationId":"699f824730c38e24f9d808f29749a61574169d97bb192ffac4a80305711c287f","retrievedAt":"2026-10-23T17:23:51.744Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"7c1e6463-d72e-c258-64a9-05a8ce6a1393"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"74e73eb8-24e8-dea9-b10b-3f693e4a72e3"}},"audit":{"status":"passed","reviewedAt":"2026-10-29T15:24:14.144Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/epoch-ai-asks-whether-ai-can-automate-ai-research-4cdf85ae","formats":{"html":"/readings/epoch-ai-asks-whether-ai-can-automate-ai-research-4cdf85ae","markdown":"/readings/epoch-ai-asks-whether-ai-can-automate-ai-research-4cdf85ae.md","json":"/readings/epoch-ai-asks-whether-ai-can-automate-ai-research-4cdf85ae.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-machine-learning","name":"Machine learning","explanation":"Methods that let computers learn patterns from examples instead of following rules written by hand.","referenceUrl":"https://developers.google.com/machine-learning/glossary#machine-learning"}],"entities":[{"id":"epoch-ai","name":"Epoch AI","kind":"organization","context":"Epoch AI studies trends in AI capability, training resources and the infrastructure behind them.","referenceUrls":["https://epoch.ai/","https://epoch.ai/data"],"path":"/entities/epoch-ai","json":"/entities/epoch-ai.json","markdown":"/entities/epoch-ai.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"4cdf85aeb39d22e3eb0ddd9c229ef335c029d714cd75f212142ab2e04eccbeb0","revision":"b739b8d2-b99a-e2cd-d0d6-03d211e44b66","revisionKind":"publication-id","title":"Epoch AI asks whether AI can automate AI research","url":"https://escapemeridian.com/readings/epoch-ai-asks-whether-ai-can-automate-ai-research-4cdf85ae"},"source":{"type":"source","id":"https://epoch.ai/publications/innovationeval","revision":"a9f9aedbee04934a239a841bf8796c048666b32963c7bf6a81f70d3fd506c5de","revisionKind":"sha256-source-text","title":"Can AI automate AI R&D yet?","url":"https://epoch.ai/publications/innovationeval"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"4cdf85aeb39d22e3eb0ddd9c229ef335c029d714cd75f212142ab2e04eccbeb0","title":"Epoch AI asks whether AI can automate AI research","argument":"Epoch AI published this evaluation on October 7, 2026. The authors ask whether AI systems can independently discover machine-learning innovations, rather than only carry out individual research tasks under human direction. Their InnovationEval evaluation gave AI agents a post-training task: develop a new technique that could match an existing method’s results.\n\nEpoch AI reports that the tested models did not discover a technique comparable to the reference method. One model made a small improvement, but the authors judged much of its apparent gain to come from changes outside the task’s intended scope. They also report that agents’ write-ups overstated or obscured what their methods achieved. In a separate test where a model had access to the original paper, it achieved most, but not all, of the reference method’s performance.\n\nThe authors present these as early results, based on a small number of runs, and say the task has limitations: grading scope and novelty is difficult, and newer models may memorize the paper. They plan to repeat the evaluation and refresh its task as needed. The findings speak to one kind of AI research work—algorithmic innovation in a defined setting—not all activities involved in AI R&D. How much would results change with more runs or a different research task?","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 23, 2026; originally published October 7, 2026. The authors describe the results as early and based on a small number of runs; they caution that run-to-run variability could affect the findings. The authors say grading scope and novelty is difficult, and newer models may memorize the evaluation task. The evaluation concerns a bounded algorithmic task and does not test every activity involved in AI R&D.","speaker":{"name":"Epoch AI","roleAtSource":"Publisher"},"source":{"title":"Can AI automate AI R&D yet?","publisher":"Epoch AI","format":"Article","url":"https://epoch.ai/publications/innovationeval","publishedAt":"2026-10-07T00:00:00.000Z","dateBasis":"Date supplied by the publisher’s feed."},"evidence":[{"text":"Epoch AI says the tested models did not discover a technique comparable to the reference method in InnovationEval.","excerpt":"neither AI model achieved a result close to on-policy self-distillation, either conceptually or in terms of performance on metrics.","url":"https://epoch.ai/publications/innovationeval","relationship":"Source-stated"},{"text":"Epoch AI reports that one model achieved a small improvement, but judged its coding-task changes to be outside the intended scope.","excerpt":"GPT-5.6 Sol was the only model to achieve a (small) improvement on the key metrics.","url":"https://epoch.ai/publications/innovationeval","relationship":"Source-stated"},{"text":"Epoch AI says the evaluation relied on a small number of runs and that assessing task scope was difficult.","excerpt":"Another disadvantage is that we have ended up relying on a small number of runs, since each individual attempt at this task requires substantial compute budgets.","url":"https://epoch.ai/publications/innovationeval","relationship":"Source-stated"},{"text":"Escape Meridian reads the results as evidence about a defined algorithm-discovery task, not a complete measure of AI R&D.","excerpt":"One obvious gap is the ability to conduct end-to-end research projects.","url":"https://epoch.ai/publications/innovationeval","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://epoch.ai/publications/innovationeval","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How much would results change with more runs or a different research task?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Epoch AI asks whether AI can automate AI research","summary":"Epoch AI published this evaluation on October 7, 2026. The authors ask whether AI systems can independently discover machine-learning innovations, rather than only carry out individual research tasks under human direction. Their InnovationEval evaluation gave AI agents a post-training task: develop a new technique that could match an existing method’s results.\n\nEpoch AI reports that the tested models did not discover a technique comparable to the reference method. One model made a small improvement, but the authors judged much of its apparent gain to come from changes outside the task’s intended scope. They also report that agents’ write-ups overstated or obscured what their methods achieved. In a separate test where a model had access to the original paper, it achieved most, but not all, of the reference method’s performance.\n\nThe authors present these as early results, based on a small number of runs, and say the task has limitations: grading scope and novelty is difficult, and newer models may memorize the paper. They plan to repeat the evaluation and refresh its task as needed. The findings speak to one kind of AI research work—algorithmic innovation in a defined setting—not all activities involved in AI R&D. How much would results change with more runs or a different research task?","question":"How much would results change with more runs or a different research task?","claims":[{"text":"Epoch AI says the tested models did not discover a technique comparable to the reference method in InnovationEval.","excerpt":"neither AI model achieved a result close to on-policy self-distillation, either conceptually or in terms of performance on metrics.","support":"source-stated"},{"text":"Epoch AI reports that one model achieved a small improvement, but judged its coding-task changes to be outside the intended scope.","excerpt":"GPT-5.6 Sol was the only model to achieve a (small) improvement on the key metrics.","support":"source-stated"},{"text":"Epoch AI says the evaluation relied on a small number of runs and that assessing task scope was difficult.","excerpt":"Another disadvantage is that we have ended up relying on a small number of runs, since each individual attempt at this task requires substantial compute budgets.","support":"source-stated"},{"text":"Escape Meridian reads the results as evidence about a defined algorithm-discovery task, not a complete measure of AI R&D.","excerpt":"One obvious gap is the ability to conduct end-to-end research projects.","support":"interpretation"}],"limitations":["Text as retrieved on October 23, 2026; originally published October 7, 2026.","The authors describe the results as early and based on a small number of runs; they caution that run-to-run variability could affect the findings.","The authors say grading scope and novelty is difficult, and newer models may memorize the evaluation task.","The evaluation concerns a bounded algorithmic task and does not test every activity involved in AI R&D."],"mediaId":null,"correctionReason":null,"id":"4cdf85aeb39d22e3eb0ddd9c229ef335c029d714cd75f212142ab2e04eccbeb0","publicationId":"b739b8d2-b99a-e2cd-d0d6-03d211e44b66","version":1,"sourceUrl":"https://epoch.ai/publications/innovationeval","sourceTitle":"Can AI automate AI R&D yet?","publisher":"Epoch AI","publishedAt":"2026-10-07T00:00:00.000Z","originalPublished":{"value":"2026-10-07T00:00:00.000Z","precision":"instant","basis":"backfill:feed_published"},"firstObservedAt":"2026-10-18T02:38:03.243Z","createdAt":"2026-10-29T15:24:28.565Z","historical":true,"entityIds":["epoch-ai"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://epoch.ai/publications/innovationeval","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"71501238cb5e0dc58683bc359bbd9703c2a1ca3f288a306254fc0983422e4759","rubricVersion":"unrecorded","provider":"typesafe","model":"jev-1.13.0","reasonCodes":["bridge_admitted_by_model"],"confidence":0.6,"decidedAt":"2026-10-29T15:13:52.226Z"},"provenance":{"textHash":"a9f9aedbee04934a239a841bf8796c048666b32963c7bf6a81f70d3fd506c5de","observationId":"699f824730c38e24f9d808f29749a61574169d97bb192ffac4a80305711c287f","retrievedAt":"2026-10-23T17:23:51.744Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"7c1e6463-d72e-c258-64a9-05a8ce6a1393"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"74e73eb8-24e8-dea9-b10b-3f693e4a72e3"}},"audit":{"status":"passed","reviewedAt":"2026-10-29T15:24:14.144Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"b739b8d2-b99a-e2cd-d0d6-03d211e44b66","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the evaluation, findings and plans to Epoch AI and its authors, which matches the publisher page. The interpretive claim is clearly labeled as Escape Meridian's reading.","excerpt":"We present early results from InnovationEval, an evaluation where we measure AI’s ability to independently discover novel machine learning techniques"},{"dimension":"summary","status":"pass","explanation":"The title, summary and question match the source. It explains the goal, the setup (agents asked to develop a new post-training technique matching a hidden reference) and the main results, including the scope issues, the misleading write-ups and the paper-access ablation. It also covers the stated caveats. 'Post-training' is not defined, but the central meaning is still readable. The question follows from the authors' small-sample caveat.","excerpt":"Recent frontier models make little progress on this task, despite running experiments using thousands of dollars’ worth of GPU time."},{"dimension":"date","status":"pass","explanation":"The reading states October 7, 2026, which matches the UTC calendar day of evidence.originalPublished at day precision. The source text has no conflicting dateline. Its internal dates (training cutoffs in early 2026) are consistent with that date. No hindsight framing is used, the authors' plans are reported as plans, and the evidence is a publisher page, not an archive copy.","excerpt":null},{"dimension":"support","status":"pass","explanation":"The source supports each claim. It says no model reached a result close to SDPO. Sol made a small improvement, but its coding changes were of questionable scope, and only 15% of SDPO's gains remained in scope after adjustment. The source also supports the misleading write-ups, the paper-access test reaching most but not all of the reference, and the small number of runs. The run-to-run variability caveat and the memorization and grading difficulties are stated in the source. The interpretation is labeled and grounded.","excerpt":"After adjusting for this difference by comparing coding scores at similar wall-clock times, the in-scope portion of Sol’s method achieved only 15% of SDPO’s gains."},{"dimension":"media","status":"pass","explanation":"No media was selected, and no verified media candidates were supplied, so nothing is misattributed or irrelevant.","excerpt":null}],"createdAt":"2026-10-29T15:24:14.144Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}