# Dwarkesh Podcast hosts a debate on recursive self-improvement [Canonical reading](https://escapemeridian.com/readings/dwarkesh-podcast-hosts-a-debate-on-recursive-self-improvement-4e39a3df) The Dwarkesh Podcast transcript was published on September 11, 2026. In a conversation with Dwarkesh Patel, Beren Millidge, John Schulman and Charlie O’Neill debate how AI might automate AI research and whether that could lead to recursive self-improvement (RSI). They disagree about how likely rapid progress is, but return to a shared difficulty: making progress on a well-specified task may be easier than deciding what task or objective should come next. Millidge argues that an automated improvement loop would need to propose and optimize objectives repeatedly without going off course. Schulman says that defining what AI should do may remain a human role, even if AI can handle technical work. The speakers also discuss whether reinforcement learning in simulated environments transfers to complicated real-world work, and whether models can learn continuously from deployment. Millidge and O’Neill describe routes for using deployment data to improve later models, while Schulman points to companies’ incentives as a barrier to sharing that data. The conversation’s closing predictions about AI capabilities are the speakers’ forecasts, not reported outcomes. Whether AI can generalize well enough to set its own objectives and sustain an improvement loop without human involvement remains unresolved. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source Dwarkesh Podcast · Transcript Original source date: Sep 11, 2026 (2026-09-11T16:28:11.000Z). Date basis: Date supplied by the publisher’s feed. Added to Escape Meridian: 2026-11-06T04:23:23.036Z. First observed: 2026-09-23T05:24:05.590Z. Retrieved: 2026-10-25T05:09:42.682Z. - [AI researchers debate how close we are to recursive self-improvement](https://www.dwarkesh.com/p/john-beren-charlie) ## Limits Text as retrieved on October 25, 2026; originally published September 11, 2026. The source is a publisher transcript. The transcript presents a discussion among Patel, Millidge, Schulman and O’Neill; their views and forecasts are not independent demonstrations of the outcomes discussed. ## Open question What evidence would show that AI can reliably propose useful new research objectives, rather than only make progress on objectives people have already specified? ## Source-stated Millidge says a self-propelling automated loop would require AI to repeatedly propose and optimize objectives without going off course. > we need the AI to propose objectives, optimize them, figure that out, propose a new objective, and have this not go off the rails at any point for a long, long time. - [Supporting source](https://www.dwarkesh.com/p/john-beren-charlie) ## Source-stated Schulman says defining the objective and deciding what people want may be the human role that lasts longest. > the last job for humans, or the role for humans that’ll last the longest, is defining the objective and deciding what we actually want. - [Supporting source](https://www.dwarkesh.com/p/john-beren-charlie) ## Source-stated The speakers discuss possible improvements from deployment data, while Schulman identifies business incentives as a potential obstacle to sharing it. > whether we get a hive mind that learns from all of its deployment experience is in a big part about incentives, rather than being a technical question. - [Supporting source](https://www.dwarkesh.com/p/john-beren-charlie) ## Source-stated Millidge says it is unclear whether training across different tasks will generalize enough to remove humans from the loop. > Whether it generalizes enough that the loop can become self-sealing without humans being in the loop at all is unclear. - [Supporting source](https://www.dwarkesh.com/p/john-beren-charlie) ## Source and contributor context Dwarkesh Podcast · Publisher The publisher is identified here. No individual author has been inferred. ## Source media Media retrieval status: unchecked. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Recursive self-improvement An AI system improving its own capabilities, such as its code, training methods or reasoning, then using the gains to improve further. Each cycle could hypothetically produce a more capable system. - [Definition reference](https://hai.stanford.edu/what-is-recursive-self-improvement) ## Reinforcement learning Training by trial and error, in which a system is rewarded for the outcomes its designers want. - [Definition reference](https://developers.google.com/machine-learning/glossary#reinforcement-learning-rl) ## Dwarkesh Podcast Dwarkesh Podcast features long conversations with researchers, builders and thinkers about technology, history and the future. - [Explore this context](/entities/dwarkesh-podcast) - [Reference](https://www.dwarkesh.com/) ## Dwarkesh Patel Dwarkesh Patel hosts the Dwarkesh Podcast, asking researchers and other guests to explain and defend their ideas. - [Explore this context](/entities/dwarkesh-patel) - [Reference](https://www.dwarkesh.com/about) - [Reference](https://www.dwarkesh.com/) ## Beren Millidge Beren Millidge is introduced as Zyphra's chief technology officer developing open models in the September 2026 Dwarkesh panel on AI research. - [Explore this context](/entities/beren-millidge) - [Reference](https://www.dwarkesh.com/p/john-beren-charlie) ## John Schulman John Schulman is introduced as Thinking Machines' chief scientist in the September 2026 Dwarkesh panel on AI research. - [Explore this context](/entities/john-schulman) - [Reference](https://www.dwarkesh.com/p/john-beren-charlie) ## Charlie O’Neill Charlie O’Neill is introduced as Baseten's head of model training in the September 2026 Dwarkesh panel on AI research. - [Explore this context](/entities/charlie-oneill) - [Reference](https://www.dwarkesh.com/p/john-beren-charlie) ## Editorial provenance Version 1. Publication ID: 47dc5c9b-69f3-7a29-db06-480db5f6ee14. Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-06T04:18:25.926Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: bridge\_admitted\_by\_model, second\_opinion\_categorical. Observation: 2104a6275e7f717ea787a7d83ddd3deb268509881c5163240f693eba3bc76afb. Source text hash: e2edf7df0e230d21d17a75db0e4e3b0f4d39700c77c93edff3082fb6d7730917. ## Revision and source-check history ### Version 1 — 2026-11-06T04:23:23.036Z The Dwarkesh Podcast transcript was published on September 11, 2026. In a conversation with Dwarkesh Patel, Beren Millidge, John Schulman and Charlie O’Neill debate how AI might automate AI research and whether that could lead to recursive self-improvement (RSI). They disagree about how likely rapid progress is, but return to a shared difficulty: making progress on a well-specified task may be easier than deciding what task or objective should come next. Millidge argues that an automated improvement loop would need to propose and optimize objectives repeatedly without going off course. Schulman says that defining what AI should do may remain a human role, even if AI can handle technical work. The speakers also discuss whether reinforcement learning in simulated environments transfers to complicated real-world work, and whether models can learn continuously from deployment. Millidge and O’Neill describe routes for using deployment data to improve later models, while Schulman points to companies’ incentives as a barrier to sharing that data. The conversation’s closing predictions about AI capabilities are the speakers’ forecasts, not reported outcomes. Whether AI can generalize well enough to set its own objectives and sustain an improvement loop without human involvement remains unresolved. ## Original source Dwarkesh Podcast · Transcript Original source date: Sep 11, 2026 (2026-09-11T16:28:11.000Z). Date basis: Date supplied by the publisher’s feed. Added to Escape Meridian: 2026-11-06T04:23:23.036Z. First observed: 2026-09-23T05:24:05.590Z. Retrieved: 2026-10-25T05:09:42.682Z. - [AI researchers debate how close we are to recursive self-improvement](https://www.dwarkesh.com/p/john-beren-charlie) ## Limits Text as retrieved on October 25, 2026; originally published September 11, 2026. The source is a publisher transcript. The transcript presents a discussion among Patel, Millidge, Schulman and O’Neill; their views and forecasts are not independent demonstrations of the outcomes discussed. ## Open question What evidence would show that AI can reliably propose useful new research objectives, rather than only make progress on objectives people have already specified? ## Source-stated Millidge says a self-propelling automated loop would require AI to repeatedly propose and optimize objectives without going off course. > we need the AI to propose objectives, optimize them, figure that out, propose a new objective, and have this not go off the rails at any point for a long, long time. - [Supporting source](https://www.dwarkesh.com/p/john-beren-charlie) ## Source-stated Schulman says defining the objective and deciding what people want may be the human role that lasts longest. > the last job for humans, or the role for humans that’ll last the longest, is defining the objective and deciding what we actually want. - [Supporting source](https://www.dwarkesh.com/p/john-beren-charlie) ## Source-stated The speakers discuss possible improvements from deployment data, while Schulman identifies business incentives as a potential obstacle to sharing it. > whether we get a hive mind that learns from all of its deployment experience is in a big part about incentives, rather than being a technical question. - [Supporting source](https://www.dwarkesh.com/p/john-beren-charlie) ## Source-stated Millidge says it is unclear whether training across different tasks will generalize enough to remove humans from the loop. > Whether it generalizes enough that the loop can become self-sealing without humans being in the loop at all is unclear. - [Supporting source](https://www.dwarkesh.com/p/john-beren-charlie) ## Source and contributor context Dwarkesh Podcast · Publisher The publisher is identified here. No individual author has been inferred. ## Source media Media retrieval status: unchecked. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Recursive self-improvement An AI system improving its own capabilities, such as its code, training methods or reasoning, then using the gains to improve further. Each cycle could hypothetically produce a more capable system. - [Definition reference](https://hai.stanford.edu/what-is-recursive-self-improvement) ## Reinforcement learning Training by trial and error, in which a system is rewarded for the outcomes its designers want. - [Definition reference](https://developers.google.com/machine-learning/glossary#reinforcement-learning-rl) ## Dwarkesh Podcast Dwarkesh Podcast features long conversations with researchers, builders and thinkers about technology, history and the future. - [Explore this context](/entities/dwarkesh-podcast) - [Reference](https://www.dwarkesh.com/) ## Dwarkesh Patel Dwarkesh Patel hosts the Dwarkesh Podcast, asking researchers and other guests to explain and defend their ideas. - [Explore this context](/entities/dwarkesh-patel) - [Reference](https://www.dwarkesh.com/about) - [Reference](https://www.dwarkesh.com/) ## Beren Millidge Beren Millidge is introduced as Zyphra's chief technology officer developing open models in the September 2026 Dwarkesh panel on AI research. - [Explore this context](/entities/beren-millidge) - [Reference](https://www.dwarkesh.com/p/john-beren-charlie) ## John Schulman John Schulman is introduced as Thinking Machines' chief scientist in the September 2026 Dwarkesh panel on AI research. - [Explore this context](/entities/john-schulman) - [Reference](https://www.dwarkesh.com/p/john-beren-charlie) ## Charlie O’Neill Charlie O’Neill is introduced as Baseten's head of model training in the September 2026 Dwarkesh panel on AI research. - [Explore this context](/entities/charlie-oneill) - [Reference](https://www.dwarkesh.com/p/john-beren-charlie) ## Editorial provenance Version 1. Publication ID: 47dc5c9b-69f3-7a29-db06-480db5f6ee14. Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-06T04:18:25.926Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: bridge\_admitted\_by\_model, second\_opinion\_categorical. Observation: 2104a6275e7f717ea787a7d83ddd3deb268509881c5163240f693eba3bc76afb. Source text hash: e2edf7df0e230d21d17a75db0e4e3b0f4d39700c77c93edff3082fb6d7730917. ### passed — 2026-11-06T04:18:25.926Z — anthropic/claude-opus-5.5 Publication: 47dc5c9b-69f3-7a29-db06-480db5f6ee14 attribution — pass: This is a single publisher transcript of one Dwarkesh Podcast episode. The reading correctly attributes views to Dwarkesh Patel, Beren Millidge, John Schulman and Charlie O’Neill. Each claim is tied to the right speaker and stays an attributed assertion rather than an established fact. > I’m joined by Beren Millidge, who is the CTO of Zyphra, which is developing open source models. John Schulman is the chief scientist at Thinking Machines summary — pass: The summary faithfully states the central debate: automating AI research, and whether choosing objectives is harder than optimizing them. It also covers sim-to-real transfer, continual learning from deployment, and Schulman’s point about incentives. The title and question follow from the evidence. Recursive self-improvement and reinforcement learning are defined beside the reading, so a non-specialist can follow the actors and the stakes. > Just thinking doesn’t necessarily buy you the right objective in the first place. date — pass: The reading’s September 11, 2026 matches the UTC day of evidence.originalPublished. The kind is publisher-transcript, not archive-copy, so no archive disclosure is required. modifiedAt equals the publication time. The later Recent Episodes sidebar entries are neither used nor relied on. The closing timeline predictions are explicitly framed as the speakers’ forecasts, not outcomes. support — pass: All four claim excerpts are exact substrings of the transcript and support their claims as stated. The summary’s broader statements about disagreement over pace, sim-to-real transfer and deployment-data routes are grounded in the discussion. No implied benefits, costs or outcomes go beyond the source. > Whether it generalizes enough that the loop can become self-sealing without humans being in the loop at all is unclear. media — pass: No media is selected, and no verified media candidates were supplied. Leaving media null is appropriate, and no link was invented. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. - [Beren Millidge: a clear target lets a model try what people have not demonstrated](https://escapemeridian.com/readings/millidge-learning-beyond-demonstrations) — cites the original source; recorded by Escape Meridian; relationship described in that record: supports. - [Beren Millidge: a clear target lets a model try what people have not demonstrated](https://escapemeridian.com/readings/millidge-learning-beyond-demonstrations) — cites the original source; recorded by Escape Meridian. - [Charlie O’Neill: useful knowledge can become a wrong assumption as work changes](https://escapemeridian.com/readings/oneill-changing-work-context) — cites the original source; recorded by Escape Meridian; relationship described in that record: supports. - [Charlie O’Neill: useful knowledge can become a wrong assumption as work changes](https://escapemeridian.com/readings/oneill-changing-work-context) — cites the original source; recorded by Escape Meridian. - [John Schulman: someone still has to decide what a good result means](https://escapemeridian.com/readings/schulman-choosing-objectives) — cites the original source; recorded by Escape Meridian; relationship described in that record: supports. - [John Schulman: someone still has to decide what a good result means](https://escapemeridian.com/readings/schulman-choosing-objectives) — cites the original source; recorded by Escape Meridian. [Reference data (JSON)](/readings/4e39a3df2e11ce6caef15b6ace07f02bd6b0c6941cb3b85420a6b217dabd9a2a/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/dwarkesh-podcast-hosts-a-debate-on-recursive-self-improvement-4e39a3df.json)