Escape Meridian

TechCrunch · Jul 6, 2024

TechCrunch reports how tokenization can limit generative AI

Markdown · JSON · Original source

TechCrunch’s article, published July 6, 2024, argues that can help explain generative AI models’ strange behaviors and stubborn limitations. Transformers process tokens—pieces of text—instead of raw text, and differences in spacing or capitalization can change how text is represented and the results a model produces. The article also describes unequal effects across languages: it reports research linking less token-efficient languages to worse performance and higher usage costs, since many vendors charge per token. It connects inconsistent digit tokenization to difficulty with numerical patterns, and says tokenization can also affect anagrams and reversing words. These examples make the representation of text a possible contributor to models’ limitations, not an established sole cause. Sheridan Feucht points to MambaByte, which works directly with raw bytes rather than tokenization, as an early research approach. Feucht says this is computationally infeasible for transformers, whose computation scales quadratically with sequence length. The article’s conclusion is that, absent a tokenization breakthrough, new model architectures may be key; whether they can make direct processing practical remains unknown.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

TechCrunchArticle · techcrunch.comOpen original source

Original source

TechCrunch · Article

Source details

Original source date: Jul 6, 2024 (2024-07-06).

Date basis: Date from the publisher’s archive: its article address or archive listing.

Added to Escape Meridian: 2026-11-18T11:58:11.362Z. First observed: 2026-10-29T06:54:03.056Z. Retrieved: 2026-10-29T06:55:34.435Z.

Referenced by

Recorded citations in public readings. A reference alone does not establish agreement or influence.

No public references recorded here yet.

Reference data (JSON)

Limits

Text as retrieved on October 29, 2026; originally published July 6, 2024. The cited research and examples are summarized as the article presents them; the evidence does not independently establish their methods or findings. The article describes as one possible explanation for model limitations; it does not establish that tokenization alone causes them. The article describes MambaByte and similar approaches as being in early research stages; it does not establish their performance in broad real-world use.

Open question

Can new model architectures make direct processing of text practical while avoiding the limitations the article attributes to ?

Source-stated

TechCrunch argues that tokenization can contribute to generative AI models’ unusual behaviors and limitations.

Understanding their “token”-based internal environments may help explain some of their strange behaviors — and stubborn limitations.

Source-stated

The article says differences in tokenization can affect how models handle spacing and capitalization.

Depending on how a model is prompted — with “once upon a” or “once upon a ,” — the results may be completely different

Source-stated

The article reports that Jun’s analysis found some languages needed up to 10 times more tokens to capture the same meaning in English.

Jun showed that some languages needed up to 10 times more tokens to capture the same meaning in English.

Source-stated

The article says vendors often charge per token, and cites research linking less token-efficient languages with worse performance and higher costs.

users of less “token-efficient” languages are likely to see worse model performance yet pay more for usage, given that many AI vendors charge per token.

Source-stated

The article reports Feucht’s view that MambaByte avoids tokenization by working directly with raw bytes, but describes such models as early research.

MambaByte, which works directly with raw bytes representing text and other data, is competitive with some transformer models on language-analyzing tasks

Source-stated

The article reports Feucht’s explanation that computation for transformer models scales quadratically with sequence length.

For transformer models in particular, computation scales quadratically with sequence length, and so we really want to use short text representations.

Source and contributor context

TechCrunch · Publisher

The publisher is identified here. No individual author has been inferred.

Source media

Media retrieval status: unchecked.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Model tokens

Small chunks of text, often parts of words, that a language model reads and writes; usage and prices are often counted in tokens.

Editorial provenance

Version 1. Publication ID: 41436ac4-7ede-df69-f38f-a46c5f5a76e7.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T11:54:26.315Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge_admitted_by_model, second_opinion_categorical.

Observation: 915615272a1bb10f09bb14549efe016500b3b2a0955e8fd2304e0bcd18bd480e. Source text hash: da43a1eb647d2634733c86880df6dec603ab04b1c6c2fd60363f8bbc0196f39b.

Revision and source-check history

Version 1 · 2026-11-18T11:58:11.362Z

TechCrunch reports how tokenization can limit generative AI

TechCrunch’s article, published July 6, 2024, argues that can help explain generative AI models’ strange behaviors and stubborn limitations. Transformers process tokens—pieces of text—instead of raw text, and differences in spacing or capitalization can change how text is represented and the results a model produces. The article also describes unequal effects across languages: it reports research linking less token-efficient languages to worse performance and higher usage costs, since many vendors charge per token. It connects inconsistent digit tokenization to difficulty with numerical patterns, and says tokenization can also affect anagrams and reversing words. These examples make the representation of text a possible contributor to models’ limitations, not an established sole cause. Sheridan Feucht points to MambaByte, which works directly with raw bytes rather than tokenization, as an early research approach. Feucht says this is computationally infeasible for transformers, whose computation scales quadratically with sequence length. The article’s conclusion is that, absent a tokenization breakthrough, new model architectures may be key; whether they can make direct processing practical remains unknown.

Original source

TechCrunch · Article

Source details

Original source date: Jul 6, 2024 (2024-07-06).

Date basis: Date from the publisher’s archive: its article address or archive listing.

Added to Escape Meridian: 2026-11-18T11:58:11.362Z. First observed: 2026-10-29T06:54:03.056Z. Retrieved: 2026-10-29T06:55:34.435Z.

Limits

Text as retrieved on October 29, 2026; originally published July 6, 2024. The cited research and examples are summarized as the article presents them; the evidence does not independently establish their methods or findings. The article describes as one possible explanation for model limitations; it does not establish that tokenization alone causes them. The article describes MambaByte and similar approaches as being in early research stages; it does not establish their performance in broad real-world use.

Open question

Can new model architectures make direct processing of text practical while avoiding the limitations the article attributes to ?

Source-stated

TechCrunch argues that tokenization can contribute to generative AI models’ unusual behaviors and limitations.

Understanding their “token”-based internal environments may help explain some of their strange behaviors — and stubborn limitations.

Source-stated

The article says differences in tokenization can affect how models handle spacing and capitalization.

Depending on how a model is prompted — with “once upon a” or “once upon a ,” — the results may be completely different

Source-stated

The article reports that Jun’s analysis found some languages needed up to 10 times more tokens to capture the same meaning in English.

Jun showed that some languages needed up to 10 times more tokens to capture the same meaning in English.

Source-stated

The article says vendors often charge per token, and cites research linking less token-efficient languages with worse performance and higher costs.

users of less “token-efficient” languages are likely to see worse model performance yet pay more for usage, given that many AI vendors charge per token.

Source-stated

The article reports Feucht’s view that MambaByte avoids tokenization by working directly with raw bytes, but describes such models as early research.

MambaByte, which works directly with raw bytes representing text and other data, is competitive with some transformer models on language-analyzing tasks

Source-stated

The article reports Feucht’s explanation that computation for transformer models scales quadratically with sequence length.

For transformer models in particular, computation scales quadratically with sequence length, and so we really want to use short text representations.

Source and contributor context

TechCrunch · Publisher

The publisher is identified here. No individual author has been inferred.

Source media

Media retrieval status: unchecked.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Model tokens

Small chunks of text, often parts of words, that a language model reads and writes; usage and prices are often counted in tokens.

Editorial provenance

Version 1. Publication ID: 41436ac4-7ede-df69-f38f-a46c5f5a76e7.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T11:54:26.315Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge_admitted_by_model, second_opinion_categorical.

Observation: 915615272a1bb10f09bb14549efe016500b3b2a0955e8fd2304e0bcd18bd480e. Source text hash: da43a1eb647d2634733c86880df6dec603ab04b1c6c2fd60363f8bbc0196f39b.

passed · 2026-11-18T11:54:26.315Z · anthropic/claude-opus-5.5

Publication: 41436ac4-7ede-df69-f38f-a46c5f5a76e7

attribution — pass: This is a single TechCrunch article, credited on the page to Kyle Wiggers. The reading attributes its arguments to TechCrunch and the article. It attributes quoted views to Sheridan Feucht and research findings to Jun and the cited studies. The reading never introduces Feucht. A brief identifier, a Northeastern PhD student studying LLM interpretability, would help but is not required.

Sheridan Feucht, a PhD student studying large language model interpretability at Northeastern University, told TechCrunch

summary — pass: The reading faithfully covers the article's thesis and its main examples: spacing, case, language inequity and cost, digits, and anagrams. It also covers the MambaByte alternative and the hedged conclusion. Tokens are defined in context and in the definitions list. 'Transformer' is only implied as a model type, and 'quadratically' is not explained. Neither blocks understanding. The reading puts 'early research' in Feucht's mouth, but the article says it in its own voice. This is a minor blending, and the claim section separates the two correctly.

Barring a tokenization breakthrough, it seems new model architectures will be the key.

date — pass: The reading gives July 6, 2024, which matches originalPublished at day precision and the URL path. The page is a publisher page, not an archive copy. The 'Most Popular' sidebar contains later stories, but the reading neither uses nor relies on them. The forecast about architectures is reported as the article's view. The reading uses no hindsight framing.

support — pass: Each claim matches an exact source excerpt. The cost and performance link is attributed to cited research. The 10x figure is attributed to Jun. The quadratic-scaling remark is attributed to Feucht. The caveats about sole causation and MambaByte's early stage stay within what the evidence supports. The question follows from the article's conclusion without asserting an outcome.

Models like MambaByte are in the early research stages, however.

media — pass: The reading selects no media, and no verified media candidates were supplied. This is appropriate.

Challenge or correct this reading

Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.