Leopold Aschenbrenner · May 29, 2024
Leopold Aschenbrenner argues in Situational Awareness that superalignment needs urgent work
Markdown · JSON · Original source
Originally published May 29, 2024, Leopold Aschenbrenner argues that reliably controlling remains an unsolved but solvable technical problem. He says (RLHF), which uses human ratings to guide AI behavior, will not scale when people can no longer assess systems’ work. His proposed “default plan” combines research on oversight, generalization, and testing with using trusted, somewhat-superhuman systems to automate . Aschenbrenner’s optimism about technical progress sits beside a warning: an intelligence explosion could leave little to test safeguards as failures become far more consequential. He argues that alignment alone is not enough; security, monitoring and limits on risky training methods could provide more margin for error. His conclusion is pointed: although he considers the research problem tractable, he says the field is not prepared for the competence and costly tradeoffs the transition would demand. He says labs should be willing to devote substantial compute to alignment and compares decisions about launching more capable systems to decisions about military operations. The consequences of alignment failure remain unclear, and his proposed path depends on systems being sufficiently aligned to trust before they help with further research.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
Leopold Aschenbrenner · Article
Source details
Original source date: May 29, 2024 (2024-05-29T22:21:37+00:00).
Date basis: Date stated on the publisher’s page.
Added to Escape Meridian: 2026-11-18T12:43:11.663Z. First observed: 2026-10-30T08:49:57.906Z. Retrieved: 2026-10-30T08:51:44.420Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
Text as retrieved on October 30, 2026; originally published May 29, 2024. Aschenbrenner presents forecasts and expectations about future systems and risks; the article does not establish those outcomes. The article does not specify what evidence would be sufficient to trust automated before using it to guide more capable systems.
Open question
What evidence would justify trusting automated before relying on it to guide more capable systems?
Source-stated
Aschenbrenner argues that reliably controlling AI systems much smarter than people remains an unsolved technical problem, though he considers it solvable.
Reliably controlling AI systems much smarter than we are is an unsolved technical problem. And while it is a solvable problem
Source-stated
Aschenbrenner argues that RLHF will not scale when people can no longer understand or assess superhuman systems’ outputs.
Aligning AI systems via human supervision (as in RLHF) won’t scale to superintelligence.
Source-stated
Aschenbrenner’s proposed approach combines research bets for aligning somewhat-superhuman systems with using trusted systems to automate further alignment research.
Then, if we’re confident we can trust these systems, we’ll need to use these somewhat-superhuman systems to automate alignment research
Source-stated
Aschenbrenner argues that the transition could leave little time for safety decisions while making failures much more consequential.
We will extremely rapidly go from systems where RLHF works fine—to systems where it will totally break down. This leaves us extremely little time to iteratively discover and address ways in which our current methods will fail.
Source-stated
Aschenbrenner concludes that the field is relying too much on luck and lacks readiness for the high-stakes decisions he describes.
We’re counting way too much on luck here.
Source and contributor context
Leopold Aschenbrenner · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Superintelligence
A hypothetical AI far more capable than people across most intellectual tasks.
Reinforcement learning from human feedback
Training that rewards a model for answers people rate highly, so it learns the kind of answers people prefer.
Interpretability
Research into how a model reaches its outputs, often by examining its internal workings.
AI alignment
Research and practice aimed at making AI systems pursue the goals their builders and users intend.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 4d59ec7c-0879-1daa-6e9d-e06b7b90e3da.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T12:39:33.470Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model.
Observation: 27c10e6f1f015f306e253602e59758cfad21035d6a58944c523bff7b5878c76c. Source text hash: 7a38cecf955fef101cfb4543e8491c66166ee2b074063b325bd82bbdd6e801fa.
Revision and source-check history
Version 1 · 2026-11-18T12:43:11.663Z
Leopold Aschenbrenner argues in Situational Awareness that superalignment needs urgent work
Originally published May 29, 2024, Leopold Aschenbrenner argues that reliably controlling remains an unsolved but solvable technical problem. He says (RLHF), which uses human ratings to guide AI behavior, will not scale when people can no longer assess systems’ work. His proposed “default plan” combines research on oversight, generalization, and testing with using trusted, somewhat-superhuman systems to automate . Aschenbrenner’s optimism about technical progress sits beside a warning: an intelligence explosion could leave little to test safeguards as failures become far more consequential. He argues that alignment alone is not enough; security, monitoring and limits on risky training methods could provide more margin for error. His conclusion is pointed: although he considers the research problem tractable, he says the field is not prepared for the competence and costly tradeoffs the transition would demand. He says labs should be willing to devote substantial compute to alignment and compares decisions about launching more capable systems to decisions about military operations. The consequences of alignment failure remain unclear, and his proposed path depends on systems being sufficiently aligned to trust before they help with further research.
Original source
Leopold Aschenbrenner · Article
Source details
Original source date: May 29, 2024 (2024-05-29T22:21:37+00:00).
Date basis: Date stated on the publisher’s page.
Added to Escape Meridian: 2026-11-18T12:43:11.663Z. First observed: 2026-10-30T08:49:57.906Z. Retrieved: 2026-10-30T08:51:44.420Z.
Limits
Text as retrieved on October 30, 2026; originally published May 29, 2024. Aschenbrenner presents forecasts and expectations about future systems and risks; the article does not establish those outcomes. The article does not specify what evidence would be sufficient to trust automated before using it to guide more capable systems.
Open question
What evidence would justify trusting automated before relying on it to guide more capable systems?
Source-stated
Aschenbrenner argues that reliably controlling AI systems much smarter than people remains an unsolved technical problem, though he considers it solvable.
Reliably controlling AI systems much smarter than we are is an unsolved technical problem. And while it is a solvable problem
Source-stated
Aschenbrenner argues that RLHF will not scale when people can no longer understand or assess superhuman systems’ outputs.
Aligning AI systems via human supervision (as in RLHF) won’t scale to superintelligence.
Source-stated
Aschenbrenner’s proposed approach combines research bets for aligning somewhat-superhuman systems with using trusted systems to automate further alignment research.
Then, if we’re confident we can trust these systems, we’ll need to use these somewhat-superhuman systems to automate alignment research
Source-stated
Aschenbrenner argues that the transition could leave little time for safety decisions while making failures much more consequential.
We will extremely rapidly go from systems where RLHF works fine—to systems where it will totally break down. This leaves us extremely little time to iteratively discover and address ways in which our current methods will fail.
Source-stated
Aschenbrenner concludes that the field is relying too much on luck and lacks readiness for the high-stakes decisions he describes.
We’re counting way too much on luck here.
Source and contributor context
Leopold Aschenbrenner · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Superintelligence
A hypothetical AI far more capable than people across most intellectual tasks.
Reinforcement learning from human feedback
Training that rewards a model for answers people rate highly, so it learns the kind of answers people prefer.
Interpretability
Research into how a model reaches its outputs, often by examining its internal workings.
AI alignment
Research and practice aimed at making AI systems pursue the goals their builders and users intend.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 4d59ec7c-0879-1daa-6e9d-e06b7b90e3da.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T12:39:33.470Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model.
Observation: 27c10e6f1f015f306e253602e59758cfad21035d6a58944c523bff7b5878c76c. Source text hash: 7a38cecf955fef101cfb4543e8491c66166ee2b074063b325bd82bbdd6e801fa.
uncertain · 2026-11-18T12:39:33.470Z · anthropic/claude-opus-5.5
Publication: 4d59ec7c-0879-1daa-6e9d-e06b7b90e3da
attribution — pass: The reading is a single essay by Leopold Aschenbrenner, part of his Situational Awareness series. The reading consistently attributes the arguments, forecasts and recommendations to him rather than presenting them as established fact.
I did spend the past year working on technical research on aligning AI systems as my day-job at OpenAI
summary — uncertain: The summary is faithful and keeps Aschenbrenner's mix of optimism and alarm, but it leaves out what he means by an intelligence explosion. That term is central to his worry and is not in the definitions list. He means a jump from roughly human-level to vastly superhuman systems, perhaps in under a year. Without that, a reader cannot see why time to test safeguards would shrink. Smallest repair: add one clause giving his definition. Superalignment in the title is adequately conveyed by the first sentence.
that we might make the transition from roughly human-level systems to vastly superhuman systems extremely rapidly, perhaps in less than a year
date — pass: The reading gives May 29, 2024, which is the UTC calendar day of originalPublished. The page is a publisher page, so no archive disclosure is needed. Forecasts are attributed as his expectations, and there is no hindsight framing. The page was modified in October 2026, but the cited passages are plainly part of the original essay body.
By the time the decade is out, we’ll have billions of vastly superhuman AI agents running around.
support — pass: The source text supports each claim, including: - RLHF failing to scale; - the default plan; - the time pressure; - extra defenses; - labs devoting compute to alignment; - the military-operation comparison; - the closing line about luck. The statement that consequences are unclear matches his own wording. The question follows from his stated uncertainty about trusting AI-produced alignment research. The cited text is plainly original essay content despite the later modification date.
Labs should be willing to commit a large fraction of their compute to automated alignment research (vs. automated capabilities research) during the intelligence explosion, if necessary.
media — pass: No media was selected and no verified media candidates were supplied, so there is no wrong or invented media to check.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A hypothetical AI far more capable than people across most intellectual tasks.
Definition referenceTraining that rewards a model for answers people rate highly, so it learns the kind of answers people prefer.
Definition referenceResearch into how a model reaches its outputs, often by examining its internal workings.
Definition referenceResearch and practice aimed at making AI systems pursue the goals their builders and users intend.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEResearch and practice aimed at making AI systems pursue the goals their builders and users intend.
Definition referenceResearch and practice aimed at making AI systems pursue the goals their builders and users intend.
Definition referenceA hypothetical AI far more capable than people across most intellectual tasks.
Definition referenceTraining that rewards a model for answers people rate highly, so it learns the kind of answers people prefer.
Definition referenceResearch into how a model reaches its outputs, often by examining its internal workings.
Definition referenceResearch and practice aimed at making AI systems pursue the goals their builders and users intend.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEResearch and practice aimed at making AI systems pursue the goals their builders and users intend.
Definition referenceResearch and practice aimed at making AI systems pursue the goals their builders and users intend.
Definition reference