# Google DeepMind reports limits of the Transformer architecture

[Canonical reading](https://escapemeridian.com/readings/google-deepmind-reports-limits-of-the-transformer-architecture-cc05c477)

Published February 24, 2024, Binghui Peng, Srini Narayanan, and Christos Papadimitriou report an argument about why large language models can hallucinate. Using Communication Complexity, they say a Transformer layer cannot compose functions when the functions’ domains are large enough; they also say examples show this inability at quite small domain sizes. Their example is identifying someone’s grandparent from a genealogy. The authors further argue that several mathematical tasks considered central to compositional tasks thought hard for large language models are unlikely to be solvable by Transformers on large enough instances, assuming certain well accepted conjectures in Computational Complexity are true. The abstract links a limitation of the architecture to tasks requiring composition, but does not establish that this is the sole or general cause of hallucinations. It gives no details of the examples or a measure of how often the limitation appears in practice. The scope of the mathematical result depends in part on the stated conjecture assumption.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

Google DeepMind · Article

Original source date: Feb 24, 2024 (2024-02-24T00:00:00+00:00).

Date basis: Date stated on the publisher’s page.

Added to Escape Meridian: 2026-11-21T11:43:10.040Z. First observed: 2026-10-28T11:51:55.229Z. Retrieved: 2026-10-28T11:53:51.735Z.

- [On Limitations of the Transformer Architecture](https://deepmind.google/research/publications/77946/)

## Limits

Text as retrieved on October 28, 2026; originally published February 24, 2024. The abstract does not identify the examples or provide details about their methods or results. The abstract does not establish how often the described inability occurs in practical use or whether it accounts for hallucinations generally. The mathematical-task claim is conditional on certain well accepted conjectures in Computational Complexity being true.


## Open question

Which tasks, beyond the genealogy example, show the reported inability at small domain sizes, and how often does it occur in practice?


## Source-stated

Binghui Peng, Srini Narayanan, and Christos Papadimitriou say a Transformer layer cannot compose functions when their domains are large enough.

> the Transformer layer is incapable of composing functions (e.g., identify a grandparent of a person in a genealogy) if the domains of the functions are large enough

- [Supporting source](https://deepmind.google/research/publications/77946/)

## Source-stated

The authors say examples show this inability when the domains are quite small.

> we show through examples that this inability is already empirically present when the domains are quite small.

- [Supporting source](https://deepmind.google/research/publications/77946/)

## Source-stated

The authors argue that several mathematical tasks central to compositional tasks thought hard for large language models are unlikely to be solvable by Transformers on large enough instances, assuming certain well accepted conjectures in Computational Complexity are true.

> several mathematical tasks that are at the core of the so-called compositional tasks thought to be hard for LLMs are unlikely to be solvable by Transformers, for large enough instances and assuming that certain well accepted conjectures in the field of Computational Complexity are true.

- [Supporting source](https://deepmind.google/research/publications/77946/)

## Source and contributor context

Google DeepMind · Publisher

The publisher is identified here. No individual author has been inferred.


## Source media

Media retrieval status: unchecked.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Transformer architecture

The neural-network design behind most current language models; it weighs how each part of the input relates to the others.

- [Definition reference](https://developers.google.com/machine-learning/glossary#transformer)

## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## Hallucination

When an AI system states something false or made up as if it were true.

- [Definition reference](https://developers.google.com/machine-learning/glossary#hallucination)

## Google DeepMind

Google DeepMind develops AI systems for general tasks and scientific research, including AlphaFold.

- [Explore this context](/entities/google-deepmind)
- [Reference](https://deepmind.google/blog/)
- [Reference](https://deepmind.google/science/alphafold/)

## Editorial provenance

Version 1. Publication ID: 3a9971fe-a7c9-7c4d-6732-c6b304027c92.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-21T11:39:53.946Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge\_admitted\_by\_model.

Observation: d2d19df2ed468f368d248b350b2a929c0ea26f3a73d051e6ee567f1bddd0f086. Source text hash: d7e1e3a5de977777fff1258ee2160614217d7e1d681b8d6c35f86318fa80df60.


## Revision and source-check history

### Version 1 — 2026-11-21T11:43:10.040Z

Published February 24, 2024, Binghui Peng, Srini Narayanan, and Christos Papadimitriou report an argument about why large language models can hallucinate. Using Communication Complexity, they say a Transformer layer cannot compose functions when the functions’ domains are large enough; they also say examples show this inability at quite small domain sizes. Their example is identifying someone’s grandparent from a genealogy. The authors further argue that several mathematical tasks considered central to compositional tasks thought hard for large language models are unlikely to be solvable by Transformers on large enough instances, assuming certain well accepted conjectures in Computational Complexity are true. The abstract links a limitation of the architecture to tasks requiring composition, but does not establish that this is the sole or general cause of hallucinations. It gives no details of the examples or a measure of how often the limitation appears in practice. The scope of the mathematical result depends in part on the stated conjecture assumption.

## Original source

Google DeepMind · Article

Original source date: Feb 24, 2024 (2024-02-24T00:00:00+00:00).

Date basis: Date stated on the publisher’s page.

Added to Escape Meridian: 2026-11-21T11:43:10.040Z. First observed: 2026-10-28T11:51:55.229Z. Retrieved: 2026-10-28T11:53:51.735Z.

- [On Limitations of the Transformer Architecture](https://deepmind.google/research/publications/77946/)

## Limits

Text as retrieved on October 28, 2026; originally published February 24, 2024. The abstract does not identify the examples or provide details about their methods or results. The abstract does not establish how often the described inability occurs in practical use or whether it accounts for hallucinations generally. The mathematical-task claim is conditional on certain well accepted conjectures in Computational Complexity being true.


## Open question

Which tasks, beyond the genealogy example, show the reported inability at small domain sizes, and how often does it occur in practice?


## Source-stated

Binghui Peng, Srini Narayanan, and Christos Papadimitriou say a Transformer layer cannot compose functions when their domains are large enough.

> the Transformer layer is incapable of composing functions (e.g., identify a grandparent of a person in a genealogy) if the domains of the functions are large enough

- [Supporting source](https://deepmind.google/research/publications/77946/)

## Source-stated

The authors say examples show this inability when the domains are quite small.

> we show through examples that this inability is already empirically present when the domains are quite small.

- [Supporting source](https://deepmind.google/research/publications/77946/)

## Source-stated

The authors argue that several mathematical tasks central to compositional tasks thought hard for large language models are unlikely to be solvable by Transformers on large enough instances, assuming certain well accepted conjectures in Computational Complexity are true.

> several mathematical tasks that are at the core of the so-called compositional tasks thought to be hard for LLMs are unlikely to be solvable by Transformers, for large enough instances and assuming that certain well accepted conjectures in the field of Computational Complexity are true.

- [Supporting source](https://deepmind.google/research/publications/77946/)

## Source and contributor context

Google DeepMind · Publisher

The publisher is identified here. No individual author has been inferred.


## Source media

Media retrieval status: unchecked.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Transformer architecture

The neural-network design behind most current language models; it weighs how each part of the input relates to the others.

- [Definition reference](https://developers.google.com/machine-learning/glossary#transformer)

## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## Hallucination

When an AI system states something false or made up as if it were true.

- [Definition reference](https://developers.google.com/machine-learning/glossary#hallucination)

## Google DeepMind

Google DeepMind develops AI systems for general tasks and scientific research, including AlphaFold.

- [Explore this context](/entities/google-deepmind)
- [Reference](https://deepmind.google/blog/)
- [Reference](https://deepmind.google/science/alphafold/)

## Editorial provenance

Version 1. Publication ID: 3a9971fe-a7c9-7c4d-6732-c6b304027c92.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-21T11:39:53.946Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge\_admitted\_by\_model.

Observation: d2d19df2ed468f368d248b350b2a929c0ea26f3a73d051e6ee567f1bddd0f086. Source text hash: d7e1e3a5de977777fff1258ee2160614217d7e1d681b8d6c35f86318fa80df60.


### uncertain — 2026-11-21T11:39:53.946Z — anthropic/claude-opus-5.5

Publication: 3a9971fe-a7c9-7c4d-6732-c6b304027c92

attribution — pass: The claims are attributed to the named authors, Peng, Narayanan and Papadimitriou, and not stated as fact. The title credits Google DeepMind, which publishes the page; the page marks two authors as external, but the summary names all three, so attribution stays clear.

> Binghui Peng\*, Srini Narayanan, Christos Papadimitriou\*

summary — uncertain: The reading is faithful and keeps the conditional framing. Its central claim, that a Transformer layer cannot compose functions when their domains are large enough, rests on terms a general reader may not know: compose functions, domains, Communication Complexity, Computational Complexity and compositional tasks. None appears in the definitions. The grandparent example partly illustrates composition, but domain size, which drives the claim, is unexplained. Smallest repair: add term-library entries for function composition, domain, Communication Complexity and Computational Complexity.

> We use Communication Complexity to prove that the Transformer layer is incapable of composing functions

date — uncertain: The page dateline, February 24, 2024, matches evidence.originalPublished. However, the page gives the venue as arXiv, so this date may be the publisher-page listing rather than the paper's first arXiv submission, which the evidence does not show. The reading uses no later material and makes no forecast-as-outcome claim. It relies only on the abstract, which is plainly original content despite the later page edit.

> February 24, 2024

support — pass: Each claim matches the abstract verbatim, with its scope and conditions: large enough domains, an empirical presence at small domains, and dependence on well accepted conjectures. The stated limitations honestly mark what the abstract does not show and add no unsupported benefits or impacts. The cited abstract text is plainly part of the original piece, so the later page edit does not affect it.

> assuming that certain well accepted conjectures in the field of Computational Complexity are true

media — pass: No media was selected, and no verified media candidates were supplied, so nothing needs checking.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/cc05c47733667f89c3ccbbd0a56d2dff1c1fd9352a5e454a6fa27f78c406bc20/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/google-deepmind-reports-limits-of-the-transformer-architecture-cc05c477.json)
