{"id":"4bf21f18d84226a0833fd1cce429b5dc158e76f1440e43d4443bcfb29b607bb9","title":"Leopold Aschenbrenner argues in Situational Awareness that superalignment needs urgent work","argument":"Originally published May 29, 2024, Leopold Aschenbrenner argues that reliably controlling superintelligence remains an unsolved but solvable technical problem. He says Reinforcement Learning from Human Feedback (RLHF), which uses human ratings to guide AI behavior, will not scale when people can no longer assess systems’ work. His proposed “default plan” combines research on oversight, generalization, interpretability and testing with using trusted, somewhat-superhuman systems to automate alignment research. Aschenbrenner’s optimism about technical progress sits beside a warning: an intelligence explosion could leave little time to test safeguards as failures become far more consequential. He argues that alignment alone is not enough; security, monitoring and limits on risky training methods could provide more margin for error. His conclusion is pointed: although he considers the research problem tractable, he says the field is not prepared for the competence and costly tradeoffs the transition would demand. He says labs should be willing to devote substantial compute to alignment and compares decisions about launching more capable systems to decisions about military operations. The consequences of alignment failure remain unclear, and his proposed path depends on systems being sufficiently aligned to trust before they help with further research.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 30, 2026; originally published May 29, 2024. Aschenbrenner presents forecasts and expectations about future systems and risks; the article does not establish those outcomes. The article does not specify what evidence would be sufficient to trust automated alignment research before using it to guide more capable systems.","speaker":{"name":"Leopold Aschenbrenner","roleAtSource":"Publisher"},"source":{"title":"IIIc. Superalignment - SITUATIONAL AWARENESS","publisher":"Leopold Aschenbrenner","format":"Article","url":"https://situational-awareness.ai/superalignment/","publishedAt":"2024-05-29T22:21:37+00:00","dateBasis":"Date stated on the publisher’s page."},"evidence":[{"text":"Aschenbrenner argues that reliably controlling AI systems much smarter than people remains an unsolved technical problem, though he considers it solvable.","excerpt":"Reliably controlling AI systems much smarter than we are is an unsolved technical problem. And while it is a solvable problem","url":"https://situational-awareness.ai/superalignment/","relationship":"Source-stated"},{"text":"Aschenbrenner argues that RLHF will not scale when people can no longer understand or assess superhuman systems’ outputs.","excerpt":"Aligning AI systems via human supervision (as in RLHF) won’t scale to superintelligence.","url":"https://situational-awareness.ai/superalignment/","relationship":"Source-stated"},{"text":"Aschenbrenner’s proposed approach combines research bets for aligning somewhat-superhuman systems with using trusted systems to automate further alignment research.","excerpt":"Then, if we’re confident we can trust these systems, we’ll need to use these somewhat-superhuman systems to automate alignment research","url":"https://situational-awareness.ai/superalignment/","relationship":"Source-stated"},{"text":"Aschenbrenner argues that the transition could leave little time for safety decisions while making failures much more consequential.","excerpt":"We will extremely rapidly go from systems where RLHF works fine—to systems where it will totally break down. This leaves us extremely little time to iteratively discover and address ways in which our current methods will fail.","url":"https://situational-awareness.ai/superalignment/","relationship":"Source-stated"},{"text":"Aschenbrenner concludes that the field is relying too much on luck and lacks readiness for the high-stakes decisions he describes.","excerpt":"We’re counting way too much on luck here.","url":"https://situational-awareness.ai/superalignment/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://situational-awareness.ai/superalignment/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"What evidence would justify trusting automated alignment research before relying on it to guide more capable systems?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Leopold Aschenbrenner argues in Situational Awareness that superalignment needs urgent work","summary":"Originally published May 29, 2024, Leopold Aschenbrenner argues that reliably controlling superintelligence remains an unsolved but solvable technical problem. He says Reinforcement Learning from Human Feedback (RLHF), which uses human ratings to guide AI behavior, will not scale when people can no longer assess systems’ work. His proposed “default plan” combines research on oversight, generalization, interpretability and testing with using trusted, somewhat-superhuman systems to automate alignment research. Aschenbrenner’s optimism about technical progress sits beside a warning: an intelligence explosion could leave little time to test safeguards as failures become far more consequential. He argues that alignment alone is not enough; security, monitoring and limits on risky training methods could provide more margin for error. His conclusion is pointed: although he considers the research problem tractable, he says the field is not prepared for the competence and costly tradeoffs the transition would demand. He says labs should be willing to devote substantial compute to alignment and compares decisions about launching more capable systems to decisions about military operations. The consequences of alignment failure remain unclear, and his proposed path depends on systems being sufficiently aligned to trust before they help with further research.","question":"What evidence would justify trusting automated alignment research before relying on it to guide more capable systems?","claims":[{"text":"Aschenbrenner argues that reliably controlling AI systems much smarter than people remains an unsolved technical problem, though he considers it solvable.","excerpt":"Reliably controlling AI systems much smarter than we are is an unsolved technical problem. And while it is a solvable problem","support":"source-stated"},{"text":"Aschenbrenner argues that RLHF will not scale when people can no longer understand or assess superhuman systems’ outputs.","excerpt":"Aligning AI systems via human supervision (as in RLHF) won’t scale to superintelligence.","support":"source-stated"},{"text":"Aschenbrenner’s proposed approach combines research bets for aligning somewhat-superhuman systems with using trusted systems to automate further alignment research.","excerpt":"Then, if we’re confident we can trust these systems, we’ll need to use these somewhat-superhuman systems to automate alignment research","support":"source-stated"},{"text":"Aschenbrenner argues that the transition could leave little time for safety decisions while making failures much more consequential.","excerpt":"We will extremely rapidly go from systems where RLHF works fine—to systems where it will totally break down. This leaves us extremely little time to iteratively discover and address ways in which our current methods will fail.","support":"source-stated"},{"text":"Aschenbrenner concludes that the field is relying too much on luck and lacks readiness for the high-stakes decisions he describes.","excerpt":"We’re counting way too much on luck here.","support":"source-stated"}],"limitations":["Text as retrieved on October 30, 2026; originally published May 29, 2024.","Aschenbrenner presents forecasts and expectations about future systems and risks; the article does not establish those outcomes.","The article does not specify what evidence would be sufficient to trust automated alignment research before using it to guide more capable systems."],"mediaId":null,"correctionReason":null,"id":"4bf21f18d84226a0833fd1cce429b5dc158e76f1440e43d4443bcfb29b607bb9","publicationId":"4d59ec7c-0879-1daa-6e9d-e06b7b90e3da","version":1,"sourceUrl":"https://situational-awareness.ai/superalignment/","sourceTitle":"IIIc. Superalignment - SITUATIONAL AWARENESS","publisher":"Leopold Aschenbrenner","publishedAt":"2024-05-29T22:21:37+00:00","originalPublished":{"value":"2024-05-29T22:21:37+00:00","precision":"instant","basis":"backfill:article_published"},"firstObservedAt":"2026-10-30T08:49:57.906Z","createdAt":"2026-11-18T12:43:11.663Z","historical":true,"entityIds":[],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://situational-awareness.ai/superalignment/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"068bce08dd735ddc2bf8776eb9abd4db6ec8f8fcea9f9b28411f2aeeaffcfd89","rubricVersion":"unrecorded","provider":"typesafe","model":"jev-1.13.0","reasonCodes":["bridge_admitted_by_model"],"confidence":0.54,"decidedAt":"2026-10-30T08:49:56.764Z"},"provenance":{"textHash":"7a38cecf955fef101cfb4543e8491c66166ee2b074063b325bd82bbdd6e801fa","observationId":"27c10e6f1f015f306e253602e59758cfad21035d6a58944c523bff7b5878c76c","retrievedAt":"2026-10-30T08:51:44.420Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"2c07ad97-a9b8-92a2-58a5-d68d4e4b7393"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"4b24b6ea-8d9d-6a9c-b7f1-09d0f0ab86ad"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-18T12:39:33.470Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/leopold-aschenbrenner-argues-in-situational-awareness-that-4bf21f18","formats":{"html":"/readings/leopold-aschenbrenner-argues-in-situational-awareness-that-4bf21f18","markdown":"/readings/leopold-aschenbrenner-argues-in-situational-awareness-that-4bf21f18.md","json":"/readings/leopold-aschenbrenner-argues-in-situational-awareness-that-4bf21f18.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-superintelligence","name":"Superintelligence","explanation":"A hypothetical AI far more capable than people across most intellectual tasks.","referenceUrl":"https://en.wikipedia.org/wiki/Superintelligence"},{"id":"term-rlhf","name":"Reinforcement learning from human feedback","explanation":"Training that rewards a model for answers people rate highly, so it learns the kind of answers people prefer.","referenceUrl":"https://developers.google.com/machine-learning/glossary#reinforcement-learning-from-human-feedback-rlhf"},{"id":"term-interpretability","name":"Interpretability","explanation":"Research into how a model reaches its outputs, often by examining its internal workings.","referenceUrl":"https://developers.google.com/machine-learning/glossary#interpretability"},{"id":"term-ai-alignment","name":"AI alignment","explanation":"Research and practice aimed at making AI systems pursue the goals their builders and users intend.","referenceUrl":"https://en.wikipedia.org/wiki/AI_alignment"}],"entities":[{"id":"time","name":"TIME","kind":"organization","context":"TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.","referenceUrls":["https://time.com/"],"path":"/entities/time","json":"/entities/time.json","markdown":"/entities/time.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"4bf21f18d84226a0833fd1cce429b5dc158e76f1440e43d4443bcfb29b607bb9","revision":"4d59ec7c-0879-1daa-6e9d-e06b7b90e3da","revisionKind":"publication-id","title":"Leopold Aschenbrenner argues in Situational Awareness that superalignment needs urgent work","url":"https://escapemeridian.com/readings/leopold-aschenbrenner-argues-in-situational-awareness-that-4bf21f18"},"source":{"type":"source","id":"https://situational-awareness.ai/superalignment/","revision":"7a38cecf955fef101cfb4543e8491c66166ee2b074063b325bd82bbdd6e801fa","revisionKind":"sha256-source-text","title":"IIIc. Superalignment - SITUATIONAL AWARENESS","url":"https://situational-awareness.ai/superalignment/"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"4bf21f18d84226a0833fd1cce429b5dc158e76f1440e43d4443bcfb29b607bb9","title":"Leopold Aschenbrenner argues in Situational Awareness that superalignment needs urgent work","argument":"Originally published May 29, 2024, Leopold Aschenbrenner argues that reliably controlling superintelligence remains an unsolved but solvable technical problem. He says Reinforcement Learning from Human Feedback (RLHF), which uses human ratings to guide AI behavior, will not scale when people can no longer assess systems’ work. His proposed “default plan” combines research on oversight, generalization, interpretability and testing with using trusted, somewhat-superhuman systems to automate alignment research. Aschenbrenner’s optimism about technical progress sits beside a warning: an intelligence explosion could leave little time to test safeguards as failures become far more consequential. He argues that alignment alone is not enough; security, monitoring and limits on risky training methods could provide more margin for error. His conclusion is pointed: although he considers the research problem tractable, he says the field is not prepared for the competence and costly tradeoffs the transition would demand. He says labs should be willing to devote substantial compute to alignment and compares decisions about launching more capable systems to decisions about military operations. The consequences of alignment failure remain unclear, and his proposed path depends on systems being sufficiently aligned to trust before they help with further research.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 30, 2026; originally published May 29, 2024. Aschenbrenner presents forecasts and expectations about future systems and risks; the article does not establish those outcomes. The article does not specify what evidence would be sufficient to trust automated alignment research before using it to guide more capable systems.","speaker":{"name":"Leopold Aschenbrenner","roleAtSource":"Publisher"},"source":{"title":"IIIc. Superalignment - SITUATIONAL AWARENESS","publisher":"Leopold Aschenbrenner","format":"Article","url":"https://situational-awareness.ai/superalignment/","publishedAt":"2024-05-29T22:21:37+00:00","dateBasis":"Date stated on the publisher’s page."},"evidence":[{"text":"Aschenbrenner argues that reliably controlling AI systems much smarter than people remains an unsolved technical problem, though he considers it solvable.","excerpt":"Reliably controlling AI systems much smarter than we are is an unsolved technical problem. And while it is a solvable problem","url":"https://situational-awareness.ai/superalignment/","relationship":"Source-stated"},{"text":"Aschenbrenner argues that RLHF will not scale when people can no longer understand or assess superhuman systems’ outputs.","excerpt":"Aligning AI systems via human supervision (as in RLHF) won’t scale to superintelligence.","url":"https://situational-awareness.ai/superalignment/","relationship":"Source-stated"},{"text":"Aschenbrenner’s proposed approach combines research bets for aligning somewhat-superhuman systems with using trusted systems to automate further alignment research.","excerpt":"Then, if we’re confident we can trust these systems, we’ll need to use these somewhat-superhuman systems to automate alignment research","url":"https://situational-awareness.ai/superalignment/","relationship":"Source-stated"},{"text":"Aschenbrenner argues that the transition could leave little time for safety decisions while making failures much more consequential.","excerpt":"We will extremely rapidly go from systems where RLHF works fine—to systems where it will totally break down. This leaves us extremely little time to iteratively discover and address ways in which our current methods will fail.","url":"https://situational-awareness.ai/superalignment/","relationship":"Source-stated"},{"text":"Aschenbrenner concludes that the field is relying too much on luck and lacks readiness for the high-stakes decisions he describes.","excerpt":"We’re counting way too much on luck here.","url":"https://situational-awareness.ai/superalignment/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://situational-awareness.ai/superalignment/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"What evidence would justify trusting automated alignment research before relying on it to guide more capable systems?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Leopold Aschenbrenner argues in Situational Awareness that superalignment needs urgent work","summary":"Originally published May 29, 2024, Leopold Aschenbrenner argues that reliably controlling superintelligence remains an unsolved but solvable technical problem. He says Reinforcement Learning from Human Feedback (RLHF), which uses human ratings to guide AI behavior, will not scale when people can no longer assess systems’ work. His proposed “default plan” combines research on oversight, generalization, interpretability and testing with using trusted, somewhat-superhuman systems to automate alignment research. Aschenbrenner’s optimism about technical progress sits beside a warning: an intelligence explosion could leave little time to test safeguards as failures become far more consequential. He argues that alignment alone is not enough; security, monitoring and limits on risky training methods could provide more margin for error. His conclusion is pointed: although he considers the research problem tractable, he says the field is not prepared for the competence and costly tradeoffs the transition would demand. He says labs should be willing to devote substantial compute to alignment and compares decisions about launching more capable systems to decisions about military operations. The consequences of alignment failure remain unclear, and his proposed path depends on systems being sufficiently aligned to trust before they help with further research.","question":"What evidence would justify trusting automated alignment research before relying on it to guide more capable systems?","claims":[{"text":"Aschenbrenner argues that reliably controlling AI systems much smarter than people remains an unsolved technical problem, though he considers it solvable.","excerpt":"Reliably controlling AI systems much smarter than we are is an unsolved technical problem. And while it is a solvable problem","support":"source-stated"},{"text":"Aschenbrenner argues that RLHF will not scale when people can no longer understand or assess superhuman systems’ outputs.","excerpt":"Aligning AI systems via human supervision (as in RLHF) won’t scale to superintelligence.","support":"source-stated"},{"text":"Aschenbrenner’s proposed approach combines research bets for aligning somewhat-superhuman systems with using trusted systems to automate further alignment research.","excerpt":"Then, if we’re confident we can trust these systems, we’ll need to use these somewhat-superhuman systems to automate alignment research","support":"source-stated"},{"text":"Aschenbrenner argues that the transition could leave little time for safety decisions while making failures much more consequential.","excerpt":"We will extremely rapidly go from systems where RLHF works fine—to systems where it will totally break down. This leaves us extremely little time to iteratively discover and address ways in which our current methods will fail.","support":"source-stated"},{"text":"Aschenbrenner concludes that the field is relying too much on luck and lacks readiness for the high-stakes decisions he describes.","excerpt":"We’re counting way too much on luck here.","support":"source-stated"}],"limitations":["Text as retrieved on October 30, 2026; originally published May 29, 2024.","Aschenbrenner presents forecasts and expectations about future systems and risks; the article does not establish those outcomes.","The article does not specify what evidence would be sufficient to trust automated alignment research before using it to guide more capable systems."],"mediaId":null,"correctionReason":null,"id":"4bf21f18d84226a0833fd1cce429b5dc158e76f1440e43d4443bcfb29b607bb9","publicationId":"4d59ec7c-0879-1daa-6e9d-e06b7b90e3da","version":1,"sourceUrl":"https://situational-awareness.ai/superalignment/","sourceTitle":"IIIc. Superalignment - SITUATIONAL AWARENESS","publisher":"Leopold Aschenbrenner","publishedAt":"2024-05-29T22:21:37+00:00","originalPublished":{"value":"2024-05-29T22:21:37+00:00","precision":"instant","basis":"backfill:article_published"},"firstObservedAt":"2026-10-30T08:49:57.906Z","createdAt":"2026-11-18T12:43:11.663Z","historical":true,"entityIds":[],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://situational-awareness.ai/superalignment/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"068bce08dd735ddc2bf8776eb9abd4db6ec8f8fcea9f9b28411f2aeeaffcfd89","rubricVersion":"unrecorded","provider":"typesafe","model":"jev-1.13.0","reasonCodes":["bridge_admitted_by_model"],"confidence":0.54,"decidedAt":"2026-10-30T08:49:56.764Z"},"provenance":{"textHash":"7a38cecf955fef101cfb4543e8491c66166ee2b074063b325bd82bbdd6e801fa","observationId":"27c10e6f1f015f306e253602e59758cfad21035d6a58944c523bff7b5878c76c","retrievedAt":"2026-10-30T08:51:44.420Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"2c07ad97-a9b8-92a2-58a5-d68d4e4b7393"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"4b24b6ea-8d9d-6a9c-b7f1-09d0f0ab86ad"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-18T12:39:33.470Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"4d59ec7c-0879-1daa-6e9d-e06b7b90e3da","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading is a single essay by Leopold Aschenbrenner, part of his Situational Awareness series. The reading consistently attributes the arguments, forecasts and recommendations to him rather than presenting them as established fact.","excerpt":"I did spend the past year working on technical research on aligning AI systems as my day-job at OpenAI"},{"dimension":"summary","status":"uncertain","explanation":"The summary is faithful and keeps Aschenbrenner's mix of optimism and alarm, but it leaves out what he means by an intelligence explosion. That term is central to his worry and is not in the definitions list. He means a jump from roughly human-level to vastly superhuman systems, perhaps in under a year. Without that, a reader cannot see why time to test safeguards would shrink. Smallest repair: add one clause giving his definition. Superalignment in the title is adequately conveyed by the first sentence.","excerpt":"that we might make the transition from roughly human-level systems to vastly superhuman systems extremely rapidly, perhaps in less than a year"},{"dimension":"date","status":"pass","explanation":"The reading gives May 29, 2024, which is the UTC calendar day of originalPublished. The page is a publisher page, so no archive disclosure is needed. Forecasts are attributed as his expectations, and there is no hindsight framing. The page was modified in October 2026, but the cited passages are plainly part of the original essay body.","excerpt":"By the time the decade is out, we’ll have billions of vastly superhuman AI agents running around."},{"dimension":"support","status":"pass","explanation":"The source text supports each claim, including:\n- RLHF failing to scale;\n- the default plan;\n- the time pressure;\n- extra defenses;\n- labs devoting compute to alignment;\n- the military-operation comparison;\n- the closing line about luck.\nThe statement that consequences are unclear matches his own wording. The question follows from his stated uncertainty about trusting AI-produced alignment research. The cited text is plainly original essay content despite the later modification date.","excerpt":"Labs should be willing to commit a large fraction of their compute to automated alignment research (vs. automated capabilities research) during the intelligence explosion, if necessary."},{"dimension":"media","status":"pass","explanation":"No media was selected and no verified media candidates were supplied, so there is no wrong or invented media to check.","excerpt":null}],"createdAt":"2026-11-18T12:39:33.470Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}