· Feb 24, 2024
Google DeepMind reports limits of the Transformer architecture
Markdown · JSON · Original source
Published February 24, 2024, Binghui Peng, Srini Narayanan, and Christos Papadimitriou report an argument about why can . Using Communication Complexity, they say a Transformer layer cannot compose functions when the functions’ domains are large enough; they also say examples show this inability at quite small domain sizes. Their example is identifying someone’s grandparent from a genealogy. The authors further argue that several mathematical tasks considered central to compositional tasks thought hard for large language models are unlikely to be solvable by Transformers on large enough instances, assuming certain well accepted conjectures in Computational Complexity are true. The abstract links a limitation of the architecture to tasks requiring composition, but does not establish that this is the sole or general cause of hallucinations. It gives no details of the examples or a measure of how often the limitation appears in practice. The scope of the mathematical result depends in part on the stated conjecture assumption.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
· Article
Source details
Original source date: Feb 24, 2024 (2024-02-24T00:00:00+00:00).
Date basis: Date stated on the publisher’s page.
Added to Escape Meridian: 2026-11-21T11:43:10.040Z. First observed: 2026-10-28T11:51:55.229Z. Retrieved: 2026-10-28T11:53:51.735Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
Text as retrieved on October 28, 2026; originally published February 24, 2024. The abstract does not identify the examples or provide details about their methods or results. The abstract does not establish how often the described inability occurs in practical use or whether it accounts for generally. The mathematical-task claim is conditional on certain well accepted conjectures in Computational Complexity being true.
Open question
Which tasks, beyond the genealogy example, show the reported inability at small domain sizes, and how often does it occur in practice?
Source-stated
Binghui Peng, Srini Narayanan, and Christos Papadimitriou say a Transformer layer cannot compose functions when their domains are large enough.
the Transformer layer is incapable of composing functions (e.g., identify a grandparent of a person in a genealogy) if the domains of the functions are large enough
Source-stated
The authors say examples show this inability when the domains are quite small.
we show through examples that this inability is already empirically present when the domains are quite small.
Source-stated
The authors argue that several mathematical tasks central to compositional tasks thought hard for large language models are unlikely to be solvable by Transformers on large enough instances, assuming certain well accepted conjectures in Computational Complexity are true.
several mathematical tasks that are at the core of the so-called compositional tasks thought to be hard for LLMs are unlikely to be solvable by Transformers, for large enough instances and assuming that certain well accepted conjectures in the field of Computational Complexity are true.
Source and contributor context
Google DeepMind · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Transformer architecture
The neural-network design behind most current language models; it weighs how each part of the input relates to the others.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Hallucination
When an AI system states something false or made up as if it were true.
Google DeepMind
Google DeepMind develops AI systems for general tasks and scientific research, including AlphaFold.
Editorial provenance
Version 1. Publication ID: 3a9971fe-a7c9-7c4d-6732-c6b304027c92.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-21T11:39:53.946Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model.
Observation: d2d19df2ed468f368d248b350b2a929c0ea26f3a73d051e6ee567f1bddd0f086. Source text hash: d7e1e3a5de977777fff1258ee2160614217d7e1d681b8d6c35f86318fa80df60.
Revision and source-check history
Version 1 · 2026-11-21T11:43:10.040Z
Google DeepMind reports limits of the Transformer architecture
Published February 24, 2024, Binghui Peng, Srini Narayanan, and Christos Papadimitriou report an argument about why can . Using Communication Complexity, they say a Transformer layer cannot compose functions when the functions’ domains are large enough; they also say examples show this inability at quite small domain sizes. Their example is identifying someone’s grandparent from a genealogy. The authors further argue that several mathematical tasks considered central to compositional tasks thought hard for large language models are unlikely to be solvable by Transformers on large enough instances, assuming certain well accepted conjectures in Computational Complexity are true. The abstract links a limitation of the architecture to tasks requiring composition, but does not establish that this is the sole or general cause of hallucinations. It gives no details of the examples or a measure of how often the limitation appears in practice. The scope of the mathematical result depends in part on the stated conjecture assumption.
Original source
· Article
Source details
Original source date: Feb 24, 2024 (2024-02-24T00:00:00+00:00).
Date basis: Date stated on the publisher’s page.
Added to Escape Meridian: 2026-11-21T11:43:10.040Z. First observed: 2026-10-28T11:51:55.229Z. Retrieved: 2026-10-28T11:53:51.735Z.
Limits
Text as retrieved on October 28, 2026; originally published February 24, 2024. The abstract does not identify the examples or provide details about their methods or results. The abstract does not establish how often the described inability occurs in practical use or whether it accounts for generally. The mathematical-task claim is conditional on certain well accepted conjectures in Computational Complexity being true.
Open question
Which tasks, beyond the genealogy example, show the reported inability at small domain sizes, and how often does it occur in practice?
Source-stated
Binghui Peng, Srini Narayanan, and Christos Papadimitriou say a Transformer layer cannot compose functions when their domains are large enough.
the Transformer layer is incapable of composing functions (e.g., identify a grandparent of a person in a genealogy) if the domains of the functions are large enough
Source-stated
The authors say examples show this inability when the domains are quite small.
we show through examples that this inability is already empirically present when the domains are quite small.
Source-stated
The authors argue that several mathematical tasks central to compositional tasks thought hard for large language models are unlikely to be solvable by Transformers on large enough instances, assuming certain well accepted conjectures in Computational Complexity are true.
several mathematical tasks that are at the core of the so-called compositional tasks thought to be hard for LLMs are unlikely to be solvable by Transformers, for large enough instances and assuming that certain well accepted conjectures in the field of Computational Complexity are true.
Source and contributor context
Google DeepMind · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Transformer architecture
The neural-network design behind most current language models; it weighs how each part of the input relates to the others.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Hallucination
When an AI system states something false or made up as if it were true.
Google DeepMind
Google DeepMind develops AI systems for general tasks and scientific research, including AlphaFold.
Editorial provenance
Version 1. Publication ID: 3a9971fe-a7c9-7c4d-6732-c6b304027c92.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-21T11:39:53.946Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_admitted_by_model.
Observation: d2d19df2ed468f368d248b350b2a929c0ea26f3a73d051e6ee567f1bddd0f086. Source text hash: d7e1e3a5de977777fff1258ee2160614217d7e1d681b8d6c35f86318fa80df60.
uncertain · 2026-11-21T11:39:53.946Z · anthropic/claude-opus-5.5
Publication: 3a9971fe-a7c9-7c4d-6732-c6b304027c92
attribution — pass: The claims are attributed to the named authors, Peng, Narayanan and Papadimitriou, and not stated as fact. The title credits Google DeepMind, which publishes the page; the page marks two authors as external, but the summary names all three, so attribution stays clear.
Binghui Peng*, Srini Narayanan, Christos Papadimitriou*
summary — uncertain: The reading is faithful and keeps the conditional framing. Its central claim, that a Transformer layer cannot compose functions when their domains are large enough, rests on terms a general reader may not know: compose functions, domains, Communication Complexity, Computational Complexity and compositional tasks. None appears in the definitions. The grandparent example partly illustrates composition, but domain size, which drives the claim, is unexplained. Smallest repair: add term-library entries for function composition, domain, Communication Complexity and Computational Complexity.
We use Communication Complexity to prove that the Transformer layer is incapable of composing functions
date — uncertain: The page dateline, February 24, 2024, matches evidence.originalPublished. However, the page gives the venue as arXiv, so this date may be the publisher-page listing rather than the paper's first arXiv submission, which the evidence does not show. The reading uses no later material and makes no forecast-as-outcome claim. It relies only on the abstract, which is plainly original content despite the later page edit.
February 24, 2024
support — pass: Each claim matches the abstract verbatim, with its scope and conditions: large enough domains, an empirical presence at small domains, and dependence on well accepted conjectures. The stated limitations honestly mark what the abstract does not show and add no unsupported benefits or impacts. The cited abstract text is plainly part of the original piece, so the later page edit does not affect it.
assuming that certain well accepted conjectures in the field of Computational Complexity are true
media — pass: No media was selected, and no verified media candidates were supplied, so nothing needs checking.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Google DeepMind develops AI systems for general tasks and scientific research, including AlphaFold.
Explore Google DeepMindA language model with many learned numerical settings, used to process and generate language.
Definition referenceWhen an AI system states something false or made up as if it were true.
Definition referenceGoogle DeepMind develops AI systems for general tasks and scientific research, including AlphaFold.
Explore Google DeepMindWhen an AI system states something false or made up as if it were true.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceWhen an AI system states something false or made up as if it were true.
Definition referenceGoogle DeepMind develops AI systems for general tasks and scientific research, including AlphaFold.
Explore Google DeepMindWhen an AI system states something false or made up as if it were true.
Definition reference