Scott Alexander argues in Astral Codex Ten that neuralese recurrence raises a taboo question

Scott Alexander argues in Astral Codex Ten that neuralese recurrence raises a taboo question

Scott Alexander’s piece, published September 24, 2026, asks whether OpenAI’s use of recurrence crosses a safety boundary around neuralese. He distinguishes chain-of-thought, where a model writes intermediate reasoning as text, from neuralese, where internal representations loop through additional processing without being written out. Alexander says Astra uses loops to add simulated layers, but this is not true neuralese, which would let an AI reason without chain-of-thought; he says nobody has invented that technology in real life. Alexander argues that loops may let a model do more processing between visible reasoning steps, potentially allowing dangerous planning to go unobserved. Repeated use of the same layers can help, he says, but does not provide the same capabilities gain as adding real layers. His central argument is that a taboo needs a clear boundary, or exceptions can weaken it. He suggests tabooing loops, setting a maximum number of real or simulated layers, or agreeing on the components of true neuralese. He concludes that AI company leaders could formalize a taboo, which he argues might be maintainable through voluntary commitments. The scale of any safety risk and whether companies will adopt such a taboo remain unknown.

Publication: 853460ae-2ef8-310c-fa8c-74cb826ed931

attribution — pass: This is a single essay by Scott Alexander on Astral Codex Ten. The reading attributes its arguments and proposals to him and does not present them as established fact.

Linch says: there’s an emerging taboo on creating the dangerous sort of neuralese AIs seen in AI 2027.

summary — uncertain: Astra is never identified as OpenAI's model, so readers must infer what it is. The reading also omits the main argument the essay responds to: OpenAI chief scientist Jakub Pachocki's defense that Astra's depth is within a factor of two of GPT-4, so its recurrence is no worse than adding normal layers. Alexander concedes this may not matter in real life but argues it matters for maintaining a taboo. Smallest repair: name Astra as OpenAI's looped model and add one sentence stating Pachocki's position and Alexander's taboo-based reply.

Pachocki says that Astra’s recurrence barely gives it any more layers than the competition.

date — pass: The originalPublished instant, 2026-09-24T22:12:39Z, falls on the UTC calendar day of September 24, 2026, matching the source dateline. The reading uses no later events and no hindsight framing. The voluntary-commitment idea is presented as Alexander's suggestion, not an outcome. modifiedAt equals the publication time.

Sep 24, 2026

support — pass: The following claims are grounded in the text: the distinction between looped transformers and true neuralese, the claim that loops allow plotting between chain-of-thought steps, the smaller gain from simulated layers than from real ones, the three proposed taboos, and the voluntary-commitment suggestion. The stated unknowns are appropriately hedged.

An AI that uses loops to simulate a large number of layers (“looped transformer”) can make a plot within a single forward pass, in between chain-of-thought steps.

media — pass: No media was selected and no verified candidates were supplied, so there is nothing to check.